Introduction to A/B testing

Analitik Pelanggan dan A/B Testing dengan Python

Ryan Grossman

Data Scientist, EDO

Overview

  • Introduction to A/B testing
  • How to design an experiment
  • Understand the logic behind A/B testing
  • Analyze the results of a test
Analitik Pelanggan dan A/B Testing dengan Python

A/B test: an experiment where you...

  • Test two or more variants against each other
  • to evaluate which one performs "best",
  • in the context of a randomized experiment
Analitik Pelanggan dan A/B Testing dengan Python

Control and treatment groups

Testing two or more ideas against each other:

  • Control: The current state of your product
  • Treatment(s): The variant(s) that you want to test

Analitik Pelanggan dan A/B Testing dengan Python

A/B Test - improving our app paywall

Question: Which paywall has a higher conversion rate?

  • Current Paywall: "I hope you enjoyed your free-trial, please consider subscribing" (control)
  • Proposed Paywall: “Your free-trial has ended, don’t miss out, subscribe today!” (treatment)

Analitik Pelanggan dan A/B Testing dengan Python

A/B testing process

  • Randomly subset the users and show one set the control and one the treatment
  • Monitor the conversion rates of each group to see which is better

Analitik Pelanggan dan A/B Testing dengan Python

The importance of randomness

  • Random assignment helps to...
    • isolate the impact of the change made
    • reduce the potential impact of confounding variables
  • Using an assignment criteria may introduce confounders
Analitik Pelanggan dan A/B Testing dengan Python

A/B testing flexibility

  • A/B testing can be use to...
    • improve sales within a mobile application
    • increase user interactions with a website
    • identify the impact of a medical treatment
    • optimize an assembly lines efficiency
    • and many more amazing things!
Analitik Pelanggan dan A/B Testing dengan Python

Good problems for A/B testing

  • Users are impacted individually
  • Testing changes that can directly impact their behavior
Analitik Pelanggan dan A/B Testing dengan Python

Bad problems for A/B testing

  • Cases with network effects among users
    • Challenging to segment the users into groups
    • Difficult to untangle the impact of the test
Analitik Pelanggan dan A/B Testing dengan Python

Let's practice!

Analitik Pelanggan dan A/B Testing dengan Python

Preparing Video For Download...