Backpropagation

Introduction to Deep Learning in Python

Dan Becker

Data Scientist and contributor to Keras and TensorFlow libraries

Backpropagation

Backpropagation

Backpropagation

Backpropagation

• Allows gradient descent to update all weights in neural network (by getting gradients for all weights)
• Comes from chain rule of calculus
• Important to understand the process, but you will generally use a library that implements this

Backpropagation process

• Trying to estimate the slope of the loss function w.r.t each weight
• Do forward propagation to calculate predictions and errors

Backpropagation process

Backpropagation process

Backpropagation process

• Go back one layer at a time
• Gradients for weight is product of:
  1. Node value feeding into that weight
  2. Slope of loss function w.r.t node it feeds into
  3. Slope of activation function at the node it feeds into

ReLU Activation Function

Backpropagation process

• Need to also keep track of the slopes of the loss function w.r.t node values
• Slope of node values are the sum of the slopes for all weights that come out of them

Let's practice!

Introduction to Deep Learning in Python