Activation functions

Introduction to Deep Learning with Keras

Miguel Esteban

Data Scientist & Founder

Introduction to Deep Learning with Keras

Introduction to Deep Learning with Keras

Introduction to Deep Learning with Keras

Introduction to Deep Learning with Keras

Introduction to Deep Learning with Keras

Effects of activation functions

Introduction to Deep Learning with Keras

Introduction to Deep Learning with Keras

Introduction to Deep Learning with Keras

Which activation function to use?

  • No magic formula
  • Different properties
  • Depends on our problem
  • Goal to achieve in a given layer
  • ReLU are a good first choice
  • Sigmoids not recommended for deep models
  • Tune with experimentation

Introduction to Deep Learning with Keras

Comparing activation functions

# Set a random seed
np.random.seed(1)

# Return a new model with the given activation
def get_model(act_function):
    model = Sequential()
    model.add(Dense(4, input_shape=(2,), activation=act_function))
    model.add(Dense(1, activation='sigmoid'))
    return model
Introduction to Deep Learning with Keras

Comparing activation functions

# Activation functions to try out
activations = ['relu', 'sigmoid', 'tanh']

# Dictionary to store results
activation_results = {}

for funct in activations:
    model = get_model(act_function=funct)
    history = model.fit(X_train, y_train,
                        validation_data=(X_test, y_test),
                        epochs=100, verbose=0)
    activation_results[funct] = history
Introduction to Deep Learning with Keras

Comparing activation functions

import pandas as pd

# Extract val_loss history of each activation function
val_loss_per_funct = {k:v.history['val_loss'] for k,v in activation_results.items()}


# Turn the dictionary into a pandas dataframe
val_loss_curves = pd.DataFrame(val_loss_per_funct)


# Plot the curves
val_loss_curves.plot(title='Loss per Activation function')
Introduction to Deep Learning with Keras

Let's practice!

Introduction to Deep Learning with Keras

Preparing Video For Download...