Calculating KPIs - a practical example

Customer Analytics and A/B Testing in Python

Ryan Grossman

Data Scientist, EDO

Goal - comparing our KPIs

Goal: Examine the KPI "user conversion rate" after the free trial
Week One Conversion Rate: Limit to users who convert in their first week after the trial ends

Conversion rate : maximum lapse date

import pandas as pd 
from datetime import datetime, timedelta

current_date = pd.to_datetime('2018-03-17')

Lapse Date: Date the trial ends for a given user

# What is the maximum lapse date in our data
print(sub_data_demo.lapse_date.max())

'2018-03-17'

KPI calculation : restrict users by lapse date

# latest lapse date: a week before today
max_lapse_date = current_date - timedelta(days=7)

# restrict to users lapsed before max_lapse_date
conv_sub_data = sub_data_demo[(sub_data_demo.lapse_date < max_lapse_date)]

# count the users remaining in our data
total_users_count = conv_sub_data.price.count()
print(total_users_count)

KPI calculation: restrict subscription date

# latest subscription date: within 7 days of lapsing
max_sub_date = conv_sub_data.lapse_date + timedelta(days=7)



# filter the users with non-zero subscription price
# who subscribed before max_sub_date
total_subs = conv_sub_data[
    (conv_sub_data.price > 0) & 
    (conv_sub_data.subscription_date <= max_sub_date)
]

# count the users remaining in our data
total_subs_count = total_subs.price.count()
print(total_subs_count)

KPI calculation: find the conversion rate

Conversion Rate: Total Subscribers / Potential Subscribers

# calculate the conversion rate with our previous values
conversion_rate = total_subs_count / total_users_count

print(conversion_rate)

0.23250807319698599

Cohort conversion rate

# Create a copy of our dataframe 
conv_sub_data = conv_sub_data.copy()

# keep users who lapsed prior to the last 14 days (2 weeks)
max_lapse_date = current_date - timedelta(days=14)
conv_sub_data = sub_data_demo[
    (sub_data_demo.lapse_date <= max_lapse_date)
]

Cohort conversion rate

Sub Time: How long it took a user to subscribe

# Find the days between lapse and subscription if they 
# subscribed ... and pd.NaT otherwise
sub_time = np.where(

        # if: a subscription date exists 
            conv_sub_data.subscription_date.notnull(),
        # then: find how many days since their lapse 
            (conv_sub_data.subscription_date - conv_sub_data.lapse_date).dt.days, 
        # else: set the value to pd.NaT   
            pd.NaT)

# create a new column 'sub_time'    
conv_sub_data['sub_time'] = sub_time

Cohort conversion rate

gcr7(), gcr14(): calculate the 7 and 14 day conversion rates

# group by the relevant cohorts
purchase_cohorts = conv_sub_data.groupby(by=['gender', 'device'], as_index=False)


# find the conversion rate for each cohort using gcr7,gcr14
purchase_cohorts.agg({sub_time: [gcr7,gcr14]})

    gender    device    sub_time
            gcr7    gcr14
0    F    and    0.221963    0.230140
1    F    iOS    0.229310    0.237931
2    M    and    0.252349    0.257718
3    M    iOS    0.218045    0.225564

How to choose KPI metrics?

Infinitely many potential KPIs
How long does it take to determine
- Monthly Conversion Rate = 1 Month Wait time
Leverage Exploratory Data Analysis
- Reveals relationships between metrics and key results
Keep In Mind How do these KPIs and my Business goals relate

Why is conversion rate important?

Strong measure of growth
Potential early warning sign of problems
- Sensitive to changes in the overall ecosystem

Next chapter: continue exploring conversion rates

How does this KPI evolve over time?
See how changes can impact different groups differently

Let's practice!

Customer Analytics and A/B Testing in Python