Performance evaluation

Fraud Detection in Python

Charlotte Werger

Data Scientist

Accuracy isn't everything

Throw accuracy out of the window when working on fraud detection problems

False positives, false negatives, and actual fraud caught

Precision-recall tradeoff

Obtaining performance metrics

# Import the packages
from sklearn.metrics import precision_recall_curve
from sklearn.metrics import average_precision_score

# Calculate average precision and the PR curve
average_precision = average_precision_score(y_test, predicted)

# Obtain precision and recall
precision, recall, _ = precision_recall_curve(y_test, predicted)

Precision-recall Curve

ROC curve to compare algorithms

# Obtain model probabilities
probs = model.predict_proba(X_test)

# Print ROC_AUC score using probabilities 
print(metrics.roc_auc_score(y_test, probs[:, 1]))

0.9338879319822626

from sklearn.metrics import classification_report, confusion_matrix

# Obtain predictions 
predicted = model.predict(X_test)

# Print classification report using predictions
print(classification_report(y_test, predicted))

  precision    recall  f1-score   support

        0.0       0.99      1.00      1.00      2099
        1.0       0.96      0.80      0.87        91

avg / total       0.99      0.99      0.99      2190

# Print confusion matrix using predictions
print(confusion_matrix(y_test, predicted))

[[2096    3]
 [  18   73]]

Let's practice!

Fraud Detection in Python