Hyperparameter tuning

Introduction to Deep Learning with Keras

Miguel Esteban

Data Scientist & Founder

Neural network hyperparameters

  • Number of layers
  • Number of neurons per layer
  • Layer order
  • Layer activations
  • Batch sizes
  • Learning rates
  • Optimizers
  • ...
Introduction to Deep Learning with Keras

Sklearn recap

# Import RandomizedSearchCV
from sklearn.model_selection import RandomizedSearchCV

# Instantiate your classifier
tree = DecisionTreeClassifier()

# Define a series of parameters to look over
params = {'max_depth':[3,None], "max_features":range(1,4), 'min_samples_leaf': range(1,4)}

# Perform random search with cross validation
tree_cv = RandomizedSearchCV(tree, params, cv=5)

tree_cv.fit(X,y)

# Print the best parameters
print(tree_cv.best_params_)

{'min_samples_leaf': 1, 'max_features': 3, 'max_depth': 3}
Introduction to Deep Learning with Keras

Turn a Keras model into a Sklearn estimator

# Function that creates our Keras model
def create_model(optimizer='adam', activation='relu'):
    model = Sequential()
    model.add(Dense(16, input_shape=(2,), activation=activation))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(optimizer=optimizer, loss='binary_crossentropy')
    return model


# Import sklearn wrapper from keras
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier


# Create a model as a sklearn estimator
model = KerasClassifier(build_fn=create_model, epochs=6, batch_size=16)
Introduction to Deep Learning with Keras

Cross-validation

# Import cross_val_score
from sklearn.model_selection import cross_val_score

# Check how your keras model performs with 5 fold crossvalidation
kfold = cross_val_score(model, X, y, cv=5)


# Print the mean accuracy per fold
kfold.mean()

0.913333

# Print the standard deviation per fold
kfold.std()

0.110754
Introduction to Deep Learning with Keras

Tips for neural networks hyperparameter tuning

  • Random search is preferred over grid search
  • Don't use many epochs
  • Use a smaller sample of your dataset
  • Play with batch sizes, activations, optimizers and learning rates
Introduction to Deep Learning with Keras

Random search on Keras models

# Define a series of parameters
params = dict(optimizer=['sgd', 'adam'], epochs=3, 
              batch_size=[5, 10, 20], activation=['relu','tanh'])

# Create a random search cv object and fit it to the data
random_search = RandomizedSearchCV(model, params_dist=params, cv=3)

random_search_results = random_search.fit(X, y)

# Print results
print("Best: %f using %s".format(random_search_results.best_score_,
random_search_results.best_params_))

Best: 0.94 using {'optimizer': 'adam', 'epochs': 3, 'batch_size': 10, 'activation': 'relu'}
Introduction to Deep Learning with Keras

Tuning other hyperparameters

def create_model(nl=1,nn=256):
    model = Sequential()
    model.add(Dense(16, input_shape=(2,), activation='relu'))

    # Add as many hidden layers as specified in nl
    for i in range(nl):
        # Layers have nn neurons
        model.add(Dense(nn, activation='relu'))
    # End defining and compiling your model...
Introduction to Deep Learning with Keras

Tuning other hyperparameters

# Define parameters, named just like in create_model()
params = dict(nl=[1, 2, 9], nn=[128,256,1000])

# Repeat the random search...

# Print results...

Best: 0.87 using {'nl': 2,'nn': 128}
Introduction to Deep Learning with Keras

Let's tune some networks!

Introduction to Deep Learning with Keras

Preparing Video For Download...