Analyse van verschil in gemiddelden A/B-tests

A/B-testen in Python

Moe Lotfy, PhD

Principal Data Science Manager

Kader voor verschil in gemiddelden

Bepaal benodigde steekproefgrootte
Voer experiment uit en doe sanity-checks

Hypothesevorming verschil in gemiddelden

Bereken metrics per variant
Analyseer het verschil met t-test

Als p-waarde < $\alpha$
- Verwerp de nulhypothese
Als p-waarde > $\alpha$
- Verwerp de nulhypothese niet

checkout.groupby('checkout_page')['time_on_page'].mean()

checkout_page
A    44.668527
B    42.723772
C    42.223772

Pingouin t-test

checkout.groupby('checkout_page')['time_on_page'].mean()

checkout_page
A    44.668527
B    42.723772
C    42.223772

ttest = pingouin.ttest(x=checkout[checkout['checkout_page']=='C']['time_on_page'], 
                       y=checkout[checkout['checkout_page']=='B']['time_on_page'],
                       paired=False,
                       alternative="two-sided")
print(ttest)

            T          dof      alternative    p-val    CI95%              cohen-d    BF10    power
T-test    -1.995423    5998    two-sided    0.046042    [-0.99, -0.01]    0.051522    0.212    0.514054

Pingouin pairwise

pairwise = pingouin.pairwise_tests(data = checkout, 

                                   dv = "time_on_page",

                                   between = "checkout_page",

                                   padjust = "bonf")

print(pairwise)

        Contrast  A  B  Paired  Parametric         T     dof alternative  \
0  checkout_page  A  B   False        True  7.026673  5998.0   two-sided   
1  checkout_page  A  C   False        True  8.833244  5998.0   two-sided   
2  checkout_page  B  C   False        True  1.995423  5998.0   two-sided   

          p-unc        p-corr p-adjust       BF10    hedges  
0  2.349604e-12  7.048812e-12     bonf  1.305e+09  0.181405  
1  1.316118e-18  3.948354e-18     bonf  1.811e+15  0.228045  
2  4.604195e-02  1.381258e-01     bonf      0.212  0.051515

Laten we oefenen!

A/B-testen in Python