Automated testing in MLOps

Fully Automated MLOps

Arturo Opsetmoen Amador

Senior Consultant - Machine Learning

What is software testing?

$$

The process of evaluating and verifying that a software product or application works as expected

$$

Three most common types of software testing:

  • Unit tests
  • Integration tests
  • End-to-end tests

$$

$$

A pyramid shoring unit testing at the base, followed by integration testing, and at the finally at the top, end-to-end tests are presented.

Fully Automated MLOps

ML software has a different nature

ML applications are not explicitly programmed

$$

A process illustrating the learning process in ML systems.

They depend on:

  • Data
  • Models
Fully Automated MLOps

Testing in an MLOps system is different

A figure showing a traditional software system that is tested with unit tests, integration tests. On the other side a ML system with data tests, model tests, and pipeline tests is shown.

1 https://research.google/pubs/pub46555/
Fully Automated MLOps

Testing ML systems

$$

In addition to traditional software testing:

  • Data tests
  • Model tests
  • Pipeline tests
Fully Automated MLOps

Testing the data

Tests for features and data include:

  • Feature expectations
  • Value from data feature justify its costs
  • Privacy control
  • Avoid used of unlawful data

$$

A set of thermometers showing different measurements is shown.

Fully Automated MLOps

Testing the models

Tests for models include:

  • Business & ML metrics correlate

  • All hyperparameters have been tuned

  • Model overfitting

  • Model staleness
  • Baseline comparison
Fully Automated MLOps

Testing ML pipelines

Tests for ML pipelines include:

  • Training is reproducible

  • Integration testing of the ML pipeline

  • Model debuggability

Fully Automated MLOps

Let's practice!

Fully Automated MLOps

Preparing Video For Download...