Metrics design and estimation

A/B Testing in Python

Moe Lotfy, PhD

Principal Data Science Manager

Types of metrics

Primary (goal/north-star):
- Best describes the success of the business or mission
Granular metrics:
- Best explain users' behavior
- More sensitive and actionable
- Signup rate:
  - = (clicks/visitors) X (signups/clicks)
Instrumentation/guardrail metrics:
- Outside the scope of this course

North star metric illustration

Bar chart and line plot metrics illustration

Types of metrics

Quantitative categorization

Means/percentiles: average sales, median time on page
Proportions:
- Signup rate: signups/total visitors
- Page abandonment rate: page abandoners/total visitors
Ratios:
- Click-through-rate(CTR): clicks/page visits or clicks/ad impressions
- Revenue per session
Metrics can be combined to form a more comprehensive success/failure criteria

Metrics requirements

Stable/robust against the unimportant differences
Sensitive to the important changes
Measurable within logging limitations
Non-gameable
- Bright colors
- Time on page

Bright exaggerated call-to-action

Python metrics estimation

checkout.groupby('gender')['purchased'].mean()

gender
F    0.908056
M    0.780009
Name: purchased, dtype: float64

checkout[(checkout['browser']=='chrome')|(checkout['browser']=='safari')]\
    .groupby('gender')['order_value'].mean()

gender
F    29.814161
M    30.383431
Name: order_value, dtype: float64

Python metrics estimation

checkout.groupby('browser')[['order_value', 'purchased']].mean()

         order_value  purchased
browser                        
chrome     30.016625   0.839088
firefox    29.887491   0.851725
safari     30.119808   0.844337

Let's practice!

A/B Testing in Python