Preparing to run an A/B test
Customer Analytics and A/B Testing in Python
Ryan Grossman
Data Scientist, EDO
A/B testing example - paywall variants
Paywall Text: Test & Control
- Current Paywall: "I hope you are enjoying the relaxing benefits of our app. Consider making a purchase."
- Proposed Paywall Don’t miss out! Try one of our new products!
Questions
- Will updating the paywall text impact our revenue?
- How do our three different consumable prices impact this?
Considerations in test design
- Can our test be run well in practice?
- Will we be able to derive meaningful results from it?
Test sensitivity
- First question: What size of impact is meaningful to detect
- 1%...?
- 20%...?
- Smaller changes = more difficult to detect
- can be hidden by randomness
- Sensitivity: The minimum level of change we want to be able to detect in our test
- Evaluate different sensitivity values
Revenue per user
# Join our demographics and purchase data purchase_data = demographics_data.merge( paywall_views,how='left', on=['uid'])# Find the total revenue per user over the period total_revenue = purchase_data.groupby(by=['uid'], as_index=False).price.sum()total_revenue.price = np.where( np.isnan(total_revenue.price), 0, total_revenue.price)# Calculate the average revenue per user avg_revenue = total_revenue.price.mean() print(avg_revenue)
16.161
Evaluating different sensitivities
avg_revenue * 1.01 # 1% lift in revenue per user
16.322839545454478
# Most reasonable option
avg_revenue * 1.1 # 10% lift in revenue per user
17.77
avg_revenue * 1.2 # 20% lift in revenue per user
19.393
Data variability
- Important to understand the variability in your data
- Does the amount spent vary a lot among users?
- If it does not then it will be easier to detect a change
Standard deviation
DataFrame.std(): Calculate the standard deviation of a pandas DataFrame
# Calculate the standard deviation of revenue per user
revenue_variation = total_revenue.price.std()
print(revenue_variation)
17.520
Variability of revenue per user
# Calculate the standard deviation of revenue per user
revenue_variation = total_revenue.price.std()
17.520
- Good to contextualize standard deviation (sd) by calculating: mean / standard deviation?
revenue_variation / avg_revenue
1.084
Variability of purchases per user
# Find the average number of purchases per user
avg_purchases = total_purchases.purchase.mean()
3.15
# Find the variance in the number of purchases per user
purchase_variation = total_purchases.purchase.std()
2.68
purchase_variation / avg_purchases
0.850
Choosing experimental unit & response variable
- Primary Goal: Increase revenue
- Better Metric: Paywall view to purchase conversion rate
- more granular than overall revenue
- directly related to the our test
- Experimental Unit: Paywall views
- simplest to work with
- assuming these interactions are independent
Finding our baseline conversion rate
- Baseline conversion rate: Conversion rate before we run the test
# Aggregate our data sets purchase_data = demographics_data.merge( paywall_views, how='inner', on=['uid'] )# conversion rate = total purchases / total paywall views conversion_rate = (sum(purchase_data.purchase) / purchase_data.purchase.count()) print(conversion_rate)
0.347
Let's practice!
Customer Analytics and A/B Testing in Python
