What is A/B testing?

A/B Testing in Python

Moe Lotfy, PhD

Principal Data Science Manager

Intro to A/B testing

  • An A/B test is...

     
    • an experiment designed to test which version is better
    • based on metric(s): signup rate, average sales per user, etc.
    • using random assignment and analyzing results
A/B Testing in Python

To A/B test or not to test?

Good use of A/B testing:

  • Optimizing conversion rates
  • Releasing new app features
  • Evaluating incremental effects of ads
  • Assessing the impact of drug trials

Do not A/B test if:

  • No sufficient traffic/"small" sample size
  • No clear logical hypothesis
  • Ethical considerations
  • High opportunity cost
A/B Testing in Python

A/B testing fundamental steps

  1. Specify the goal and designs/experiences
  2. Randomly sample users for enrollment
  3. Randomly assign users to:
    • control variant: current state
    • treatment/test variant(s): new design
  4. Log user actions and compute metrics
  5. Test for statistically significant differences

A/B testing steps

A/B Testing in Python

Value of randomization

  • Generalizability and representativeness
  • Minimizing bias between groups
  • Establishing causality by isolating treatment effect

 

Random sampling and random assignment illustration

1 https://www.statology.org/random-selection-vs-random-assignment/
A/B Testing in Python

Python example of random assignment

checkout.info()

RangeIndex: 9000 entries, 0 to 8999
Data columns (total 6 columns):
 #   Column         Non-Null Count  Dtype  
 0   user_id        9000 non-null   int64  
 1   checkout_page  9000 non-null   object 
 2   order_value    7605 non-null   float64
 3   purchased      9000 non-null   float64
 4   gender         9000 non-null   object 
 5   browser        9000 non-null   object 
dtypes: float64(2), int64(1), object(3)
memory usage: 422.0+ KB
A/B Testing in Python

Python example of random assignment

checkout['gender'].value_counts(normalize=True)

F    0.507556
M    0.492444
Name: gender, dtype: float64

sample_df = checkout.sample(n=3000)
sample_df['gender'].value_counts(normalize=True)

M    0.506333
F    0.493667
Name: gender, dtype: float64
A/B Testing in Python

Python example of random assignment

checkout.groupby('checkout_page')['gender'].value_counts(normalize=True)

checkout_page  gender
A              M         0.505000
               F         0.495000
B              F         0.507333
               M         0.492667
C              F         0.520333
               M         0.479667
Name: gender, dtype: float64
A/B Testing in Python

Let's practice!

A/B Testing in Python

Preparing Video For Download...