Model evaluation

Predicting CTR with Machine Learning in Python

Kevin Huo

Instructor

Precision and recall

  • Precision: proportion of clicks relative to total number of impressions, TP / (TP + FP)
    • Higher precision means higher ROI on ad spend
  • Recall: the proportion of clicks gotten of all clicks available, TP / (TP + FN)
    • Higher recall means better targeting of relevant audience
Predicting CTR with Machine Learning in Python

Calculating precision and recall

print(precision_score(
  y_test, y_pred, average = 'weighted'))

0.73

print(recall_score(
  y_test, y_pred, average = 'weighted'))

0.75
Predicting CTR with Machine Learning in Python

Baseline classifiers

  • It is important to evaluate classifiers relative to an appropriate baseline
    • The baseline here, due to imbalanced nature of click data, is a classifier that always predicts no click
y_pred = np.asarray([0 for x in range(len(X_test))])

[[0]
 [0] ...]
Predicting CTR with Machine Learning in Python

Implications on ROI analysis

  • For the baseline classifier, tp and fp will be zero
  • Therefore total return and total spend will be zero, and ROI undefined
  • Confusion matrix via confusion_matrix() along with ravel() to get the four categories of outcomes
    total_return = tp * r
total_spent = (tp + fp) * cost
roi = total_return / total_spent
Predicting CTR with Machine Learning in Python

Let's practice!

Predicting CTR with Machine Learning in Python

Preparing Video For Download...