Experimental design: setting up testing parameters

A/B Testing in Python

Moe Lotfy, PhD

Principal Data Science Manager

Distribution parameters

  • d follows a normal distribution Mathematical formulation of the Null and alternative hypotheses

  • If observed difference 'd' is unlikely:

    • reject the Null hypothesis
  • Null vs alternative hypothesis distributions

Graphical representation of the distributions of the Null and alternative hypotheses

A/B Testing in Python

Design parameters and error types

  • Power (1- $\beta$)
    • $\beta$ = Type II error = False negative
    • Commonly set at 80%
  • Minimum Detectable Effect (MDE)
    • Smallest difference we care to capture

Error types matrix

Graphical representation of experimental paramters

A/B Testing in Python

Design parameters and error types

  • Significance level $\alpha$
    • $\alpha$ = Type I error = False positive
    • Commonly set at 5%
  • P-value
    • Probability of obtaining a result assuming the Null hypothesis is true.
    • If p-value < $\alpha$
      • Reject Null hypothesis
    • If p-value > $\alpha$
      • Fail to reject Null hypothesis

Error types matrix

Graphical representation of experimental paramters

A/B Testing in Python

Experiment parameters analogy

Analogy for explaining statistical power and parameters:

  1. Time at store = sample size/experiment duration
  2. Bag of chips size = effect size/MDE
  3. Store cleanliness/organization = data variance

Statistical power analogy

A/B Testing in Python

Let's practice!

A/B Testing in Python

Preparing Video For Download...