Overfitting and ensembling

Machine Learning for Finance in Python

Nathan George

Data Science Professor

overfitting

Simplify your model

limited net

Neural network options

Options to combat overfitting:

Decrease number of nodes
Use L1/L2 regulariation
Dropout
Autoencoder architecture
Early stopping
Adding noise to data
Max norm constraints
Ensembling

Dropout

dropout

Dropout in keras

from keras.layers import Dense, Dropout

model = Sequential()
model.add(Dense(500,
                input_dim=scaled_train_features.shape[1],
                activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(100, activation='relu'))
model.add(Dense(1, activation='linear'))

Test set comparison

R$^2$ values on AMD without dropout:

train: 0.91
test: -0.72

With dropout:

train: 0.46
test: -0.22

Ensembling

random forest

Implementing ensembling

# make predictions from 2 neural net models
test_pred1 = model_1.predict(scaled_test_features)
test_pred2 = model_2.predict(scaled_test_features)

# horizontally stack predictions and take the average across rows
test_preds = np.mean(np.hstack((test_pred1, test_pred2)), axis=1)

Comparing the ensemble

Model 1 R$^2$ score on test set:

-0.179

model 2:

-0.148

ensemble (averaged predictions):

-0.146

Dropout and ensemble!

Machine Learning for Finance in Python