Initial A/B test design

Customer Analytics and A/B Testing in Python

Ryan Grossman

Data Scientist, EDO

Increasing our app's revenue with A/B testing

Specific Goals:

  • Test change to our consumable purchase paywall to...
  • Increase revenue by increasing the purchase rate

General Concepts:

  • A/B testing techniques transfer across a variety of contexts
  • Keep in mind how you would apply these techniques
Customer Analytics and A/B Testing in Python

Paywall views & Demographics data

demographics_data = pd.read_csv('user_demographics.csv')
demographics_data.head(n=2)

         uid              reg_date device gender country  age
0  5.277e+07  2018-03-07T00:00:00Z    and      F     FRA   27
1  8.434e+07  2017-09-22T00:00:00Z    iOS      F     TUR   22

paywall_views = pd.read_csv('paywall_views.csv')
paywall_views.head(n=2)

        uid                      date  purchase  sku  price
0  32209877 2016-12-04 14:20:49+00:00         0  NaN    NaN
1  32209877 2016-12-05 22:17:12+00:00         0  NaN    NaN
Customer Analytics and A/B Testing in Python

Chapter 3 goals

  • Introduce the foundations of A/B testing
  • Walk through the code need to apply these concepts
Customer Analytics and A/B Testing in Python

Response variable

  • The quantity used to measure the impact of your change
  • Should either be a KPI or directly related to a KPI
  • The easier to measure the better
Customer Analytics and A/B Testing in Python

Factors & variants

  • Factors: The type of variable you are changing
    • The paywall color
  • Variants: Particular changes you are testing
    • A red versus blue paywall

Customer Analytics and A/B Testing in Python

Experimental unit of our test

  • The smallest unit you are measuring the change over
  • Individual users make a convenient experimental unit
Customer Analytics and A/B Testing in Python

Calculating experimental units

# Join our paywall views to the user demographics
purchase_data = demographics_data.merge(
    paywall_views, how='left', on=['uid'])


# Find the total purchases for each user
total_purchases = purchase_data.groupby(
    by=['uid'], as_index=False).purchase.sum()

# Find the mean number of purchases per user
total_purchases.purchase.mean()

3.15
Customer Analytics and A/B Testing in Python

Calculating experimental units

# Find the minimum number of purchases made by a user
# over the period 
total_purchases.purchase.min()

0.0

# Find the maximum number of purchases made by a user
# over the period 
total_purchases.purchase.max()

17.0
Customer Analytics and A/B Testing in Python

Experimental unit of our test

User-days: User interactions on a given day

  • More convenient than users by itself
  • Not required to track user's actions across time
  • Can treat simpler actions as responses to the test
Customer Analytics and A/B Testing in Python

Calculating user-days

# Group our data by users and days, then find the total purchases
total_purchases = purchase_data.groupby(
    by=['uid', 'date'], as_index=False)).purchase.sum()


# Calcualte summary statistics across user-days
total_purchases.purchase.mean()
total_purchases.purchase.min()
total_purchases.purchase.max()

0.0346

0.0

3.0
Customer Analytics and A/B Testing in Python

Randomness of experimental units

  • Best to randomize by individuals regardless of our experimental unit
  • Otherwise users can have inconsistent experience
  • This can impact the tests results
Customer Analytics and A/B Testing in Python

Designing your A/B test

  • Good to understand the qualities of your metrics and experimental units
  • Important to build intuition about your users and data overall
Customer Analytics and A/B Testing in Python

Let's practice!

Customer Analytics and A/B Testing in Python

Preparing Video For Download...