A/B Testing

Understanding Data Science

Lis Sulmont

Curriculum Manager, DataCamp

Data science workflow

Data science workflow

Understanding Data Science

What are experiments in data science?

Experiments help drive decisions and draw conclusions

  1. Form a question
  2. Form a hypothesis
  3. Collect data
  4. Test the hypothesis with a statistical test
  5. Interpret results
Understanding Data Science

Case study: which is the better blog post title?

Form a question: Does blog title A or blog title B result in more clicks?

Form a hypothesis: Blog title A and blog title B result in the same amount of clicks.

Collect data:

  • 50% users will see blog title A
  • 50% users will see blog title B
  • Track click-through rate until sample size reached

a-b-test-headlines.jpg

Understanding Data Science

Case study: which is the better blog post title?

Test the hypothesis with a statistical test: Is the difference in titles' click-through rates significant?

Interpret results:

  • Choose a title
  • Or ask more questions and design another experiment!

a-b-test-headlines.jpg

Understanding Data Science

What is A/B Testing?

AKA Champion/Challenger Testing

a-b-test.png

Understanding Data Science

Terminology Review

  • Sample size: number of data points used
  • Statistical significance: result is likely not due to chance
    • Given assumptions of statistical model
    • Use statistical tests to calculate this:
      • e.g., t-test, Z-test, ANOVA, Chi-square test
Understanding Data Science

A/B Testing Steps

  • Picking a metric to track
  • Calculating sample size
  • Running the experiment
  • Checking for significance
Understanding Data Science

Pick a metric to track: click-through rate

click-on-link.jpg

Understanding Data Science

sample-size-percent.jpg

  • Baseline metric to gauge any changes
    • How often people generally click on a link to our blogs
  • If the rate is much larger or smaller than 50%, we need a large sample size
    • Click rate is typically small (<3%)
Understanding Data Science

sample-size-sensitivity.jpg

Larger sample sizes allow us to detect smaller changes

Understanding Data Science

Run your experiment

run-experiment.jpg

Understanding Data Science

Check for significance

significance.jpg

Understanding Data Science

What if the results aren't significant?

  • Difference is smaller than the threshold we chose
  • Running our test longer won't help
  • Still might be a difference; it's just small and insignificant to us
Understanding Data Science

Let's practice!

Understanding Data Science

Preparing Video For Download...