Performance estimation

Monitoring Machine Learning Concepts

Hakim Elakhrass

Co-founder and CEO of NannyML

Overview

Two performance estimation algorithms:

  • CBPE - classification tasks
  • DLE - regression tasks
Monitoring Machine Learning Concepts

CBPE - How it works

  • Estimates a confusion matrix

 

  • Allows to calculate classification metrics like accuracy, precision, recall

 

  • Captures the impact of covariate shift on the model

 

The CBPE algorithm is illustrated in an image that shows how it works. The first model makes a prediction based on a given example, and this prediction is used to determine the class. For instance, if the model predicts a value of 0.9 for an example, then the class is positive because the value is above 0.5. Otherwise, if the value is below 0.5, the class is negative. This prediction process is carried out for all examples in the dataset, and the resulting confidence scores are aggregated to generate an estimated confusion matrix.

Monitoring Machine Learning Concepts

CBPE - Considerations

  • No covariate shift in the unseen regions

 

  • No concept drift is present in the incoming data

 

  • Requires a probability calibration
Monitoring Machine Learning Concepts

DLE - How it works

  • Predicts the absolute error of the model

 

  • Uses an external child model

 

  • Allows to calculate various regression metrics like MAE, MSE, MSLE

 

  • Captures the presence of covariate shift in the input data

The image describes the performance estimation process for a regression task in production. Initially, the analysis data is fed into the model to generate predictions. These predictions, along with the analysis data, are then passed to the child model, which predicts the error. Using this error prediction, other regression metrics are calculated to evaluate the performance of the model.

Monitoring Machine Learning Concepts

DLE - Considerations

  • No covariate shift in the unseen regions

 

  • No concept drift is present in the incoming data

 

  • Extra complexity
Monitoring Machine Learning Concepts

Let's practice!

Monitoring Machine Learning Concepts

Preparing Video For Download...