Automated hyperparameter tuning

Fully Automated MLOps

Arturo Opsetmoen Amador

Senior Consultant - Machine Learning

What is a hyperparameter?

Hyperparameters are tunable values that control the learning process

  • Not learned during the training process
  • Set before training an ML model

$$

Examples:

  • Model architecture in a Neural Network
  • Number of branches in a decision tree
  • Learning rate
Fully Automated MLOps

What is hyperparameter tuning?

Image showing a data scientists adjusting hyperparameters in a neural network. The neural network training routine tunes parameters learnable at training time. These combinations produce different scores.

Fully Automated MLOps

Hyperparameter tuning methods

  • Grid Search

$$

$$

  • Random Search

$$

$$

  • Bayesian Optimization

Grid search animation showing orderly search.

Random search animation showing aleatory search.

Bayesian search optimization showing intelligent search.

Fully Automated MLOps

Automate hyperparameter tuning

Image showing a motor performing the exploration of hyperparameter combinations.

Fully Automated MLOps

Automated hyperparameter tuning steps

  • Need to define:
    • Set of hyperparameters to optimize
    • Search space for each parameter
    • A performance metrics to optimize
    • Stopping Criteria

An image showing a process: define hyperparameters to optimize, define the search space for eeach hyperparameter, specify the target to optimize, specify stopping criteria. At the bottom the whole process is powered by a hyperparameter tuning method.

Fully Automated MLOps

Automatically finding the best set of hyperparameters

$$

The previous process image is extended. At the bottom the best set of hyperparameters is found.

Fully Automated MLOps

Hyperparameters and environment symmetry

$$

An image illustrating environment symmetry. The same set of tuned hyperparameters are used in development and experimentation and in production.

Fully Automated MLOps

Hyperparameter tuning - Experiment tracking

An image showing the automated hyperparameter tuning process. At the top of the process, the automated experiment tracking system collects metadata such as the hyperparameters used during the tuning process. These metadata is saved to the metadata store.

Fully Automated MLOps

Example - Hyperparameter visualization

An example of a hyperparameter tuning visualization tool.

Fully Automated MLOps

Let's practice!

Fully Automated MLOps

Preparing Video For Download...