Automated experiment tracking

Fully Automated MLOps

Arturo Opsetmoen Amador

Senior Consultant - Machine Learning

Machine learning has an experimental nature

Many "levers" to experiment with:

  • Data transformations
    • Features used
  • Training algorithms
    • Hyperparameter tuning
  • Evaluation metrics
    • Precision, recall, etc.
Fully Automated MLOps

A large space of possibilities!

$$

Photo of a several antennas pointing towards a sky full of stars. The image is meant to illustrate a large space, difficult to fully explore.

Fully Automated MLOps

Problems of Manual Machine Learning Workflows

Lack of automation = problems in manual ML workflows

  • Difficult to track experiments and results
  • Waste time and resources
  • Hard to reproduce/share results

Image of a very frustrated data scientist in front of a laptop.

Fully Automated MLOps

Automated logging in ML

Important things to log include:

  • Code
  • Environment
  • Data
  • Parameters
  • Metrics
Fully Automated MLOps

The importance of logging

Logging is essential for:

  • The reproducibility of experiments in ML systems.
  • Tracking system performance and making informed decisions.
  • Identifying potential issues for improvement.

$$

Reproducibility provides transparency and is crucial to make our systems trustworthy.

Fully Automated MLOps

Automated experiment tracking system

$$

Organize logs per run, or experiment to:

  • See model training metadata
  • Compare model training runs
  • Reproduce model training runs

$$

An IT architecture figure. The experiment tracking system is made of two components, a dashboard user interface and a programmatic access interface. The dashboard UI can be accessed by the users of the system. The programmatic access is used by other architecture components. The experiment tracking system loggs and reads from other component in the system that is the one in charge of storing loggs.

Fully Automated MLOps

Automated experiment tracking - Today's market

Several tools that automate experiment tracking:

  • TensorBoard

$$

  • MLFlow

$$

  • Weights & Biases

$$

  • Neptune

$$

Tensorflow logo.

$$

MLflow logo.

$$ Weight & Biases logo.

Neptune.AI logo.

Fully Automated MLOps

Let's practice!

Fully Automated MLOps

Preparing Video For Download...