Gradient descent

Introduction to Deep Learning in Python

Dan Becker

Data Scientist and contributor to Keras and TensorFlow libraries

Gradient descent

Gradient descent

Gradient descent

Gradient descent

Gradient descent

Gradient descent

Gradient descent

Gradient descent

Gradient descent

Gradient descent

Gradient descent

Gradient descent

• If the slope is positive:
  • Going opposite the slope means moving to lower numbers
  • Subtract the slope from the current value
  • Too big a step might lead us astray
• Solution: learning rate
  • Update each weight by subtracting learning rate * slope

Slope calculation example

• To calculate the slope for a weight, need to multiply:
  • Slope of the loss function w.r.t value at the node we feed into
  • The value of the node that feeds into our weight
  • Slope of the activation function w.r.t value we feed into

Slope calculation example

• To calculate the slope for a weight, need to multiply:
  • Slope of the loss function w.r.t value at the node we feed into
  • The value of the node that feeds into our weight
  • Slope of the activation function w.r.t value we feed into

Slope calculation example

• Slope of mean-squared loss function w.r.t prediction:
  • 2 (Predicted Value - Actual Value) = 2 Error
  • 2 * -4

Slope calculation example

• To calculate the slope for a weight, need to multiply:
  • Slope of the loss function w.r.t value at the node we feed into
  • The value of the node that feeds into our weight
  • Slope of the activation function w.r.t value we feed into

Slope calculation example

• To calculate the slope for a weight, need to multiply:
  • Slope of the loss function w.r.t value at the node we feed into
  • The value of the node that feeds into our weight
  • Slope of the activation function w.r.t value we feed into

Slope calculation example

• To calculate the slope for a weight, need to multiply:
  • Slope of the loss function w.r.t value at the node we feed into
  • The value of the node that feeds into our weight
  • Slope of the activation function w.r.t value we feed into

Slope calculation example

• To calculate the slope for a weight, need to multiply:
  • Slope of the loss function w.r.t value at the node we feed into
  • The value of the node that feeds into our weight
  • Slope of the activation function w.r.t value we feed into

Slope calculation example

• 2 * -4 * 3
• -24
• If learning rate is 0.01, the new weight would be
• 2 - 0.01(-24) = 2.24

Network with two inputs affecting prediction

Code to calculate slopes and update weights

import numpy as np
weights = np.array([1, 2])
input_data = np.array([3, 4])
target = 6
learning_rate = 0.01
preds = (weights * input_data).sum()
error = preds - target

print(error)

5

Code to calculate slopes and update weights

gradient = 2 * input_data * error

gradient

array([30, 40])

weights_updated = weights - learning_rate * gradient
preds_updated = (weights_updated * input_data).sum()
error_updated = preds_updated - target

print(error_updated)

2.5

Let's practice!

Introduction to Deep Learning in Python