A/B Testing

Understanding Data Science

Lis Sulmont

Curriculum Manager, DataCamp

Data science workflow

Data science workflow

What are experiments in data science?

Experiments help drive decisions and draw conclusions

Form a question
Form a hypothesis
Collect data
Test the hypothesis with a statistical test
Interpret results

Case study: which is the better blog post title?

Form a question: Does blog title A or blog title B result in more clicks?

Form a hypothesis: Blog title A and blog title B result in the same amount of clicks.

Collect data:

50% users will see blog title A
50% users will see blog title B
Track click-through rate until sample size reached

Case study: which is the better blog post title?

Test the hypothesis with a statistical test: Is the difference in titles' click-through rates significant?

Interpret results:

Choose a title
Or ask more questions and design another experiment!

What is A/B Testing?

AKA Champion/Challenger Testing

Terminology Review

Sample size: number of data points used
Statistical significance: result is likely not due to chance
- Given assumptions of statistical model
- Use statistical tests to calculate this:
  - e.g., t-test, Z-test, ANOVA, Chi-square test

A/B Testing Steps

Picking a metric to track
Calculating sample size
Running the experiment
Checking for significance

Pick a metric to track: click-through rate

Baseline metric to gauge any changes
- How often people generally click on a link to our blogs
If the rate is much larger or smaller than 50%, we need a large sample size
- Click rate is typically small (<3%)

Larger sample sizes allow us to detect smaller changes

Run your experiment

Check for significance

What if the results aren't significant?

Difference is smaller than the threshold we chose
Running our test longer won't help
Still might be a difference; it's just small and insignificant to us

Let's practice!

Understanding Data Science

Preparing Video For Download...