Model performance measurement

Machine Learning for Business

Karolis Urbonas

Head of Machine Learning & Science, Amazon

Performance measurement types

Two key supervised learning metrics:

Accuracy --> classification
Error --> regression

Classification performance

Accuracy
Recall
Precision

Churn example

Churn prediction

Mis-classified items

Another churn prediction

Accuracy

acc

Precision

precision

Recall

recall

Regression performance

Error

Regression example

Predicting revenue with a line

Regression error

Testing non-linear models

Error improvements

Actionable models - A/B testing

Good models are not always actionable:

Churn prediction, purchase prediction, machine failure prediction

Test if using models helps improve outcomes:

Target customers predicted to churn with incentives (discounts, coupons, promotions)

Send reminder emails and product details to customers likely to purchase

_Did this result in decreased churn, increased purchase rate and reduced machine failures? If yes, build into automated process. If no, collect more data, improve models, and test again. _

Let's practice!

Machine Learning for Business