Reporting A/B test results

A/B Testing in R

Lauryn Burleigh

Data Scientist

A/B visualizations

Send conclusions outside the data team

Clear visualizations
Accurately convey findings

A/B visualizations

Bar plots

Hypothesis comparing group means
- t-test

Visualizes:

Mean
Error bars
- Standard deviation
- Standard error
- Confidence interval

A bar plot showing the mean of the time to eat Pepperoni pizza in pink and time to eat Cheese pizza in blue, on the x-axis. Error bars are plotted for each bar.

Bar data formatting

library(dplyr)

data <- pizza %>%
    group_by(Topping) %>%

    summarise(AvgTime = mean(Time), 
              SdTime = sd(Time))

data

# A tibble: 2 × 3
  Topping   AvgTime SdTime
  <chr>       <dbl>  <dbl>
1 Cheese      5.81  0.948
2 Pepperoni   6.46  1.06

Using bar plots

library(ggplot2)

ggplot(data, aes(x = Topping,

              y = AvgTime,

              fill = Topping)) +

  geom_bar(stat =  "summary", 
           fun = "mean") +

  geom_errorbar(aes(
    ymin = AvgTime - SdTime, 
    ymax = AvgTime + SdTime))

A bar plot showing the mean of the currently implemented web design in pink and new web design option in blue, on the x-axis. Error bars are plotted for each bar.

Box plots

Comparing group medians
- Mann-Whitney U
- Chi-square
- Fisher's Exact
- t-tests
Bottom box limit: 25% of data points below
Top box limit: 75% of data points below
Whiskers: highest and lowest values
- Outliers excluded
Outliers: point outside of whiskers

A box plot of the time to eat Pepperoni pizza in pink and the time to eat Cheese pizza in blue.

Using box plots

Data in long format
Ideal for researchers or statisticians or assessing non-normal distributions

ggplot(pizza, aes(x = Topping, y = Time, 
                fill = Topping)) +

  geom_boxplot()

A box plot of the time to eat Pepperoni pizza in pink and the time to eat Cheese pizza in blue.

Scatter plots

Relationship between variables
- Correlations
Every point
Trend: direction and rate

A scatter plot showing the time to eat pizza on the x-axis and enjoyment of pizza the y-axis.

Using scatter plots

library(ggplot2)

ggplot(pizza, aes(x = Time, 
                  y = Enjoyment,
                  fill = Topping)) +

  geom_point()

Scatter plot of a positive correlation with time on the x-axis and enjoyment on the y-axis with each point colored either pink if Pepperoni pizza and blue if Cheese pizza.

Using scatter plots

Subset data to assess one group

ggplot(subset(pizza, 
              Topping %in% 
              "Pepperoni"),
       aes(x = Time, y = Enjoy)) + 
  geom_point()

Scatter plot of a positive correlation with time on the x-axis and enjoyment on the y-axis of only Pepperoni pizza.

Presenting data

Audience has no statistics background:

Statistic values uninformative
Use real-world findings
Test conclusion
Power analysis
Error bars

Stock image of data results being presented to two individuals.

Presenting data example

Results:

Significant t-test
Power = 0.9

Presentation:

Significant difference was found
Pepperoni pizza was eaten quicker than Cheese
Reliable results
No large variation (error bars)

A bar plot showing the mean of the currently implemented web design in pink and new web design option in blue, on the x-axis. Error bars are plotted for each bar.

Let's practice!

A/B Testing in R