Challenges of monitoring ML models

Monitoring Machine Learning Concepts

Hakim Elakhrass

Co-founder and CEO of NannyML

Machine learning project components

A Venn diagram with three circles represents the fields of software engineering, data analytics, and machine learning, with the intersection of all three representing the field of data science.

The model fails to make predictions

Possible problems :

Language barriers - combining different programming languages using the "glue" code

Code maintenance - compatibility problem of updated dependencies

Scaling issues - not robust infrastructure to handle more users

The model predictions fail

Covariate shift

Change in the input's distribution
Possible to detect using statistical methods
Not every drift impact performance

A density distribution plot comparing the age of customers in the training and production sets reveals that the training set is heavily skewed towards lower age values. The production set is similar in shape but with a spike in higher age values.

Concept drift

Change in the relationship between the input data and targets
Difficult to detect
Almost always affects the business impact of the model

There is a relationship between the customer lifetime value (CLV) and the input feature Age for both the training and production sets. However, the relationship changes in production, which is an indication of concept drift.

Availability of ground truth

An image of a clothing store.

Let's practice!

Monitoring Machine Learning Concepts