Classification metrics

Model Validation in Python

Kasey Jones

Data Scientist

Classification metrics

  • Precision
  • Recall (also called sensitivity)
  • Accuracy
  • Specificity
  • F1-Score, and its variations
  • ...
Model Validation in Python

Classification metrics

  • Precision
  • Recall (also called sensitivity)
  • Accuracy
  • Specificity
  • F1-Score, and its variations
  • ...
Model Validation in Python

Confusion matrix

A confusion matrix is a 2x2 matrix that represents how many observations were predicted 0 or 1, and how many observations were actually 0 or 1. It can be used to calculate all kinds of accuracy metrics.

Model Validation in Python
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, test_predictions)
print(cm)

array([[ 23,  7],
       [  8, 62]])

cm[<true_category_index>, <predicted_category_index>]
cm[1, 0]

8
Model Validation in Python

Accuracy

Accuracy can be calculated by using the main diagonal of the confusion matrix.

Model Validation in Python

Precision

Precision focuses on the observations that were predicted to be 1. Of all the predictions that were 1, how many of them actually were 1?

Model Validation in Python

Recall

Recall focuses on the observations that were actually 1’s. How many of these 1’s did we predict to be a 1?

Model Validation in Python

Accuracy, precision, recall

from sklearn.metrics import accuracy_score, precision_score, recall_score
accuracy_score(y_test, test_predictions)

.85

precision_score(y_test, test_predictions)

.8986

recall_score(y_test, test_predictions)

.8857
Model Validation in Python

Practice time

Model Validation in Python

Preparing Video For Download...