Applications of metric evaluation

Predicting CTR with Machine Learning in Python

Kevin Huo

Instructor

Four categories of outcomes

Example of four categories of outcomes in classification

  • First part of category (true/false) represents whether model was correct or not
  • Second part of the category (positive/negative) represents the target label the model applied
Predicting CTR with Machine Learning in Python

Interpretations of four categories

  • If model predicts there is a click, then there is a bid for that impression which costs money
  • If no click predicted, no bidding and hence no cost
  • True positives (TP): money gained (impressions paid for that were clicked on).
  • False positives (FP): money lost (impressions that were paid for, but not clicked).
  • True negatives (TN): money saved (no click predicted so no impressions bought).
  • False negatives (FN): money lost out on (no click predicted, but would have been actual click in reality).
Predicting CTR with Machine Learning in Python

Confusion matrix

print(confusion_matrix(y_test, y_pred))

[[8163  166]
 [1517  154]]

# Order: tn, fp, fn, tp
print(confusion_matrix(y_test, y_pred).ravel())

[8163, 166, 1517, 154]
Predicting CTR with Machine Learning in Python

ROI analysis

  • Assume: some cost c and return r per X number of impressions
total_return = tp * r

total_cost = (tp + fp) * c

tp * r > (tp + fp) * c

roi = total_return / total_spent
Predicting CTR with Machine Learning in Python

Let's practice!

Predicting CTR with Machine Learning in Python

Preparing Video For Download...