Evaluating performance

Understanding Machine Learning

Hadrien Lacroix

Content Developer at DataCamp

Evaluate step

Understanding Machine Learning

Overfitting

  • Performs great on training data
  • Performs poorly on testing data
  • Model memorized training data and can't generalize learnings to new data
  • Use testing set to check model performance
Understanding Machine Learning

Illustrating overfitting

overfitting

Understanding Machine Learning

Accuracy

  • Accuracy = correctly classified observations / all observations
  • 48 / 50 = 96%

linear classifier from first lesson

Understanding Machine Learning

Limits of accuracy: fraud example

$$

Accuracy of this model:

$$\frac{28\ correctly\ classified}{30\ total\ points}=93.33\%$$

  • Misses majority of fraudulent transactions
  • Need a better metric
Understanding Machine Learning

Confusion matrix

confusion matrix

Understanding Machine Learning

True positives

Understanding Machine Learning

True positives

Understanding Machine Learning

False negatives

Understanding Machine Learning

False negatives

Understanding Machine Learning

Remembering False Negatives

fp.jpg

Understanding Machine Learning

Fill out the rest...

Understanding Machine Learning

False positives, true negatives

Understanding Machine Learning

Remembering False Positives

fn2.jpg

Understanding Machine Learning

Sensitivity

How many fraudulent transactions did we classify correctly?

$$Sensitivity = \frac{true\ positives}{true\ positives + false\ negatives} =1/3=33.33\% $$

  • Rather mark legitimate transactions as suspicious than authorize fraudulent transactions
Understanding Machine Learning

Specificity

$$Specificity = \frac{true\ negatives}{true\ negatives + false\ positives}$$

Spam filter:

  • Rather send spam to inbox than send real emails to the spam folder
Understanding Machine Learning

Evaluating regression

Understanding Machine Learning

Evaluating regression

  • Error = distance between point (actual value) and line (predicted value)
  • Many ways calculate this. e.g, root mean square error
Understanding Machine Learning

Unsupervised learning

choose your own adventure

1 https://www.flickr.com/photos/micahdowty/8540188997
Understanding Machine Learning

Let's practice!

Understanding Machine Learning

Preparing Video For Download...