How to handle concept drift?

Monitoring Machine Learning Concepts

Hakim Elakhrass

Co-founder and CEO of NannyML

Concept drift detection

Error-based methods
- tracking error changes over time
- requires ground truth

Train a new model using training and production data
- change in the predictions is a concept drift
- expensive in more advanced use-cases

Retraining

The image depicts two graphs. The first graph illustrates a gradual decrease in model accuracy over time. In contrast, the second graph shows a slight decrease in accuracy followed by a retraining process, resulting in an increase in accuracy. This process is then repeated at intervals.

Pros :

keep the model up-to-date with recent patterns

Cons :

increased costs and risk of failure
doesn't provide the root cause of the problem

Online learning

The image illustrates the cycle of online learning, where data from the app is sent to the database and immediately used to update the model. The updated model is then deployed again for further use.

Pros :

real-time adaptation to changing conditions

Cons :

requires constant access to ground truth
sensitive to noise
needs careful parameter tuning

Other resolutions

A event-specific model for reoccurring events

Weighting the importance of new data
- with most focus on newer data, model can adapt easier

The image shows a timeline where the regular model is deployed on November 23rd, and then the Black Friday model is deployed on November 24th. After that, the regular model is redeployed again.

Let's practice!

Monitoring Machine Learning Concepts