Monitoring technical performance directly

Monitoring Machine Learning Concepts

Hakim Elakhrass

Co-founder and CEO of NannyML

Covariate shift - performance relationship

Three covariate shifts:

  • Shifts to certain regions

    • more high-income people applying for loans

  • Shifts to underrepresented regions

    • having 10% of tech people applying for loans instead of 0.5% from training data

  • Shifts to less certain regions

    • shift from 20% to 40% of middle-income people which are close to decision boundary

 

-> No impact

 

-> Unknown impact

 

-> Negative impact

Monitoring Machine Learning Concepts

Guaranteed negative impact

 
 
 
 

Covariate shift to uncertain regions always negatively impacts performance

The visualization of the training period highlights the region where the model is least certain. However, during production, there are more people located within this region than before.

Monitoring Machine Learning Concepts

False alerts problem

The image shows dots representing alerts, and how important dots can get lost when there are too many dots

Monitoring Machine Learning Concepts

The importance of technical performance

  • Direct metric of how well the model performs the task at hand

 

  • Reflects any silent model failure

 

  • Removes the overload of false alerts

The images shows a piece of the monitoring diagram. Performance monitoring in the first box,under it connected with a dotted line and in a dotted box asking if models is still performing, to the right a diamond showing "performance degradation", then arrow the right with "yes" above, and an arrow looping back to the performance monitoring box, with "no" above.

Monitoring Machine Learning Concepts

Let's practice!

Monitoring Machine Learning Concepts

Preparing Video For Download...