Business value calculation and estimation

Monitoring Machine Learning in Python

Maciej Balawejder

Data Scientist

Model business value

  • The aim of machine learning model is to provide value to the business.
  • The business value of the model can decrease due to:
    • Change in customer's habits
    • The model might not be useful anymore
Monitoring Machine Learning in Python

Confusion matrix

The image shows confusion matrix with TP, FN, FP and TN values.

  • True positive (TP) - the model correctly predicts that a booking will not be canceled
  • False positive (FP) - the model incorrectly predicts a booking will not be canceled
  • False negative (FN) - the model incorrectly predicts that a booking will be canceled
  • True negative (TN) - the model correctly predicts that a booking will be canceled
Monitoring Machine Learning in Python

Business value formula

The image shows multiplication and summation of confusion matrix and business value matrix.

  • True positive (TP) - doesn't add or subtract any value.
  • False positive (FP) - leads to relocations and discounts, it costs hotel $200.
  • False negative (FN) - costs hotel $100, a one-night stay until a replacement is found.
  • True negative (TN) - worth $1000 because the hotel can rent the room to someone else.
Monitoring Machine Learning in Python

When labels are available

The image shows the process of calculating model business value when labels are available.

# Initialize the calculator
calculator = nannyml.PerformanceCalculator(...
    problem_type='classification_binary',
    metrics=['business_value'],
    # [value_of_TN, value_of_FP], [value_of_FN, value_of_TP]]
    business_value_matrix = [[0, -200],[-100, 1000]],
    normalize_business_value='None')
Monitoring Machine Learning in Python

When labels are not available

The image shows the process of calculating model business value when labels are not available.

# Initialize the estimator
estimator = nannyml.CBPE(...
    problem_type='classification_binary',
    metrics=['business_value'],
    # [value_of_TN, value_of_FP], [value_of_FN, value_of_TP]]
    business_value_matrix = [[0, -200],[-100, 1000]],
    normalize_business_value='per_prediction')
Monitoring Machine Learning in Python

Let's practice!

Monitoring Machine Learning in Python

Preparing Video For Download...