Machine learning risks

Machine Learning for Business

Karolis Urbonas

Head of Machine Learning & Science, Amazon

Poor performance

Some models perform poorly (make sure you review test performance, not training):

Low precision
Low recall
Large error

Low precision

Low precision - a lot of misclassified items in the class of interest = a lot of false positives

Example - only 10% of customers identified as likely to purchase actually purchased the product

Low recall

Low recall - only a small fraction of all observations in the class have been correctly captured (recalled) by the model

Example - only 25% of all fraudulent transactions identified by the model

Large error

Large error - large differences between predicted and actual values

Example - the average error for the customer satisfaction rating prediction is 3.5 units or 70% in percentage points

Non-actionable model use cases

Q: How to test the models correctly?

A: Run tests / experiments to validate their performance e.g. churn prevention emails, product promotions, manual machine maintenance, manual transaction review

A/B testing

abtest

What if tests don't work?

Get more data - business has to be involved
Build causal models to understand drivers
Run qualitative research (surveys etc.)
Change the scope of the problem
- Narrow
- Widen
- Different question

Let's practice!

Machine Learning for Business