A/B Testing best practices and advanced topics intro

A/B Testing in Python

Moe Lotfy, PhD

Principal Data Science Manager

Best practices

Avoid peeking

  • Avoid making decisions by peeking at the results before reaching the designed sample size, as this inflates error rates similar to multiple comparisons.

Account for day-of-the-week effects

  • Users may behave differently on weekends versus weekdays, so we should include overall behavior.
A/B Testing in Python

Best practices

  • Simplicity/feasibility:
    • Do we need to build the full feature?
    • Painted door tests
  • Isolation
    • Change one variable at a time to attribute impact.
A/B Testing in Python

Advanced topics

  • Multifactorial design and interaction effects
    • Measures the isolated effect of each variable
    • Uncovers interaction/synergistic effects
  • Bayesian A/B testing
    • Incorporates prior data into current experiment
    • Views population parameters as distributions
    • More intuitive understanding of test results
  • SUTVA violation and network effects
    • One user's assignment in a test impacts others
    • Common in social networks A/B tests
    • One solution: clusters assignment
A/B Testing in Python

Let's practice!

A/B Testing in Python

Preparing Video For Download...