Availability of ground truth

Monitoring Machine Learning Concepts

Hakim Elakhrass

Co-founder and CEO of NannyML

Instant ground truth

An image of a taxi which relates to taxi arrival prediction.

  • Measures actual performance

  • Easy evaluation

  • Accurate

Monitoring Machine Learning Concepts

Production data - instant

The graph illustrates a performance graph of a classification problems, where the "reference period" represents the testing dataset, and the "analysis period" represents a stream of production data. Additionally, there is a tabular dataset with highlighted ground truth values.

Monitoring Machine Learning Concepts

Delayed ground truth

An image of the empty wallet which relates to the loan default prediction.

  • Delay depends on the application
  • A possible scenario is loan default prediction
  • Unknown performance in the meantime
  • Requires performance estimation
Monitoring Machine Learning Concepts

Production data - delayed

The graph illustrates a performance graph of a classification problems, where the "reference period" represents the testing dataset, and the "analysis period" represents a stream of production data. Additionally, there is a tabular dataset with highlighted ground truth values.

Monitoring Machine Learning Concepts

Absent ground truth

An image of insurance policy which relates to the insurance pricing case.

  • Present in fully-automated processes
  • A possible scenario is insurance pricing
  • Actual performance is unknown
  • Requires performance estimation
Monitoring Machine Learning Concepts

Production data - absent

The graph illustrates a performance graph of a classification problems, where the "reference period" represents the testing dataset, and the "analysis period" represents a stream of production data. Additionally, there is a tabular dataset with highlighted ground truth values.

Monitoring Machine Learning Concepts

Let's practice!

Monitoring Machine Learning Concepts

Preparing Video For Download...