What is NannyML?

Monitoring Machine Learning in Python

Hakim Elakhrass

Co-founder and CEO of NannyML

Prerequisites

  • Machine Learning Monitoring Concepts topics:
    • The ideal monitoring workflow
    • Challenges of monitoring ML models in production
    • Two silent model failures, covariate shift, and concept drift
    • Six methods for detecting covariate shift
    • The theoretical concepts behind CBPE and DLE performance estimation methods
Monitoring Machine Learning in Python

What this course will cover?

  • Build a robust monitoring system using NannyML
  • Implement performance estimation algorithms
  • Use the business calculator to determine the monetary value of your machine learning model
  • Run univariate and multivariate covariate shift detection methods
Monitoring Machine Learning in Python

Monitoring challenges

$$

No access to ground truth

The graph shows the ROC AUC metric over time, where after deployment, the graph is empty since there's access to no ground truth.

$$

Alerts fatigue

The image shows data distribution changes over time; some are red, indicating a shift, with written comments. Is it time to worry, or is it a false alarm?

Monitoring Machine Learning in Python

Open-source solution

The image shows the NannyML logo and companies that use NannyML, like Walmart, Allianz, UBS, or DeepMind. Also, it shows the information that NannyML works with any framework.

Monitoring Machine Learning in Python

The key features

  1. Monitor what matter
    • Performance estimation and calculation
    • Business value estimation and calculation
  2. Find what is broken
    • Univariate
    • Multivariate
    • Data quality
  3. Fix it
    • Retraining triggers
Monitoring Machine Learning in Python

How to use NannyML?

The image shows the testing and production data coming to NannyML.

import nannyml

# Load the dataset
reference, analysis, analysis_gt = nannyml.load_us_census_ma_employment_data()
Monitoring Machine Learning in Python

Let's practice!

Monitoring Machine Learning in Python

Preparing Video For Download...