Analyzing difference in means A/B tests

A/B Testing in Python

Moe Lotfy, PhD

Principal Data Science Manager

Framework for difference in means

Calculate required sample size
Run experiment and perform sanity checks

Difference in means hypothesis formulation

Calculate the metrics per variant
Analyze the difference using t-test

If p-value < $\alpha$
- Reject Null hypothesis
If p-value > $\alpha$
- Fail to reject Null hypothesis

checkout.groupby('checkout_page')['time_on_page'].mean()

checkout_page
A    44.668527
B    42.723772
C    42.223772

Pingouin t-test

checkout.groupby('checkout_page')['time_on_page'].mean()

checkout_page
A    44.668527
B    42.723772
C    42.223772

ttest = pingouin.ttest(x=checkout[checkout['checkout_page']=='C']['time_on_page'], 
                       y=checkout[checkout['checkout_page']=='B']['time_on_page'],
                       paired=False,
                       alternative="two-sided")
print(ttest)

            T          dof      alternative    p-val    CI95%              cohen-d    BF10    power
T-test    -1.995423    5998    two-sided    0.046042    [-0.99, -0.01]    0.051522    0.212    0.514054

Pingouin pairwise

pairwise = pingouin.pairwise_tests(data = checkout, 

                                   dv = "time_on_page",

                                   between = "checkout_page",

                                   padjust = "bonf")

print(pairwise)

        Contrast  A  B  Paired  Parametric         T     dof alternative  \
0  checkout_page  A  B   False        True  7.026673  5998.0   two-sided   
1  checkout_page  A  C   False        True  8.833244  5998.0   two-sided   
2  checkout_page  B  C   False        True  1.995423  5998.0   two-sided   

          p-unc        p-corr p-adjust       BF10    hedges  
0  2.349604e-12  7.048812e-12     bonf  1.305e+09  0.181405  
1  1.316118e-18  3.948354e-18     bonf  1.811e+15  0.228045  
2  4.604195e-02  1.381258e-01     bonf      0.212  0.051515

Let's practice!

A/B Testing in Python