Course introduction and overview

Analitik Pelanggan dan A/B Testing dengan Python

Ryan Grossman

Data Scientist, EDO

What is A/B testing?

  • A/B Testing: Test different ideas against each other in the real world
  • Choose the one that statistically performs better
Analitik Pelanggan dan A/B Testing dengan Python

Why is A/B testing important?

  • No guessing
  • Provides accurate answers - quickly
  • Allows to rapidly iterate on ideas
  • ...and establish causal relationships

Analitik Pelanggan dan A/B Testing dengan Python

A/B test process

  1. Develop a hypothesis about your product or business
  2. Randomly assign users to two different groups
  3. Expose:
    • Group 1 to the the current product rules
    • Group 2 to a product that tests the hypothesis
  4. Pick whichever performs better according to a set of KPIs

Analitik Pelanggan dan A/B Testing dengan Python

Where can A/B testing be used?

Users + ideas → A/B test

  • testing impact of drugs
  • incentivizing spending
  • driving user growth
  • ...and many more!

Analitik Pelanggan dan A/B Testing dengan Python

Course progression

  1. Understanding users — Key Performance Indicators
  2. Identifying trends — Exploratory Data Analysis
  3. Optimizing performance — Design of A/B Tests
  4. Data driven decisions — Analyzing A/B Test Results
Analitik Pelanggan dan A/B Testing dengan Python

Key performance indicators (KPIs)

  • A/B Tests: Measure impact of changes on KPIs
  • KPIs — metrics important to an organization
    • likelihood of a side-effect
    • revenue
    • conversion rate
    • ...

Analitik Pelanggan dan A/B Testing dengan Python

How to identify KPIs

Experience + Domain knowledge + Exploratory data analysis

  • Experience & Knowledge - What is important to a business
  • Exploratory Analysis - What metrics and relationships impact these KPIs
Analitik Pelanggan dan A/B Testing dengan Python

Next Up...

  • Exploratory Data Analysis (EDA)
  • Identify KPIs and areas for further analysis

Analitik Pelanggan dan A/B Testing dengan Python

Let's practice!

Analitik Pelanggan dan A/B Testing dengan Python

Preparing Video For Download...