The need for optimization

Introduction to Deep Learning in Python

Dan Becker

Data Scientist and contributor to Keras and TensorFlow libraries

A baseline neural network

ch2_1_v3.002.png

Introduction to Deep Learning in Python

A baseline neural network

ch2_1_v3.003.png

Introduction to Deep Learning in Python

A baseline neural network

ch2_1_v3.004.png

Introduction to Deep Learning in Python

A baseline neural network

ch2_1_v3.005.png

Introduction to Deep Learning in Python

A baseline neural network

ch2_1_v3.006.png

Introduction to Deep Learning in Python

A baseline neural network

ch2_1_v3.007.png

Introduction to Deep Learning in Python

A baseline neural network

ch2_1_v3.008.png

Introduction to Deep Learning in Python

A baseline neural network

ch2_1_v3.009.png

Introduction to Deep Learning in Python

A baseline neural network

ch2_1_v3.010.png

Introduction to Deep Learning in Python

Predictions with multiple points

  • Making accurate predictions gets harder with more points
  • At any set of weights, there are many values of the error
  • ... corresponding to the many points we make predictions for
Introduction to Deep Learning in Python

Loss function

  • Aggregates errors in predictions from many data points into single number
  • Measure of model's predictive performance
Introduction to Deep Learning in Python

Squared error loss function

ch2_1_v3.019.png

Introduction to Deep Learning in Python

Squared error loss function

ch2_1_v3.020.png

Introduction to Deep Learning in Python

Squared error loss function

ch2_1_v3.021.png

Introduction to Deep Learning in Python

Loss function

ch2_1_v3.023.png

Introduction to Deep Learning in Python

Loss function

ch2_1_v3.024.png

Introduction to Deep Learning in Python

Loss function

ch2_1_v3.025.png

Introduction to Deep Learning in Python

Loss function

  • Lower loss function value means a better model
  • Goal: Find the weights that give the lowest value for the loss function
  • Gradient descent
Introduction to Deep Learning in Python

Gradient descent

  • Imagine you are in a pitch dark field
  • Want to find the lowest point
  • Feel the ground to see how it slopes
  • Take a small step downhill
  • Repeat until it is uphill in every direction
Introduction to Deep Learning in Python

Gradient descent steps

  • Start at random point
  • Until you are somewhere flat:
    • Find the slope
    • Take a step downhill
Introduction to Deep Learning in Python

Optimizing a model with a single weight

ch2_1_v3.042.png

Introduction to Deep Learning in Python

Optimizing a model with a single weight

ch2_1_v3.044.png

Introduction to Deep Learning in Python

Optimizing a model with a single weight

ch2_1_v3.045.png

Introduction to Deep Learning in Python

Optimizing a model with a single weight

ch2_1_v3.046.png

Introduction to Deep Learning in Python

Optimizing a model with a single weight

ch2_1_v3.047.png

Introduction to Deep Learning in Python

Optimizing a model with a single weight

ch2_1_v3.048.png

Introduction to Deep Learning in Python

Optimizing a model with a single weight

ch2_1_v3.049.png

Introduction to Deep Learning in Python

Optimizing a model with a single weight

ch2_1_v3.050.png

Introduction to Deep Learning in Python

Optimizing a model with a single weight

ch2_1_v3.051.png

Introduction to Deep Learning in Python

Let's practice!

Introduction to Deep Learning in Python

Preparing Video For Download...