A/B testing for marketing

Analyzing Marketing Campaigns with pandas

Jill Rosok

Data Scientist

What is A/B testing?

Prior to running the test determine:

What is the desired outcome of the test? What is our hypothesis?
What is the metric we are trying to impact (i.e., page views, conversions)?
Will we get enough traffic to our site to reach statistical significance and make a decision in a timely manner?

Testing allows us to understand marketing impact

How long does a test need to run?

Personalized email test

Test allocation

email = marketing[marketing['marketing_channel'] == 'Email']
allocation = email.groupby(['variant'])\
                           ['user_id'].nunique()

allocation.plot(kind='bar')
plt.title('Personalization test allocation')
plt.xticks(rotation = 0)
plt.ylabel('# participants')
plt.show()

Allocation plot

Setting up our data to evaluate the test

# Group by user_id and variant
subscribers = email.groupby(['user_id', 
                             'variant'])['converted'].max()

subscribers = pd.DataFrame(subscribers.unstack(level=1))

Setting up our data to evaluate the test

# Drop missing values from the control column
control = subscribers['control'].dropna()

# Drop missing values from the personalization column
personalization = subscribers['personalization'].dropna()

Conversion rates

print("Control conversion rate:", 
      np.mean(control))
print("Personalization conversion rate:", 
      np.mean(personalization))

Control conversion rate: 0.2814814814814815
Personalization conversion rate: 0.3908450704225352

Let's get testing!

Analyzing Marketing Campaigns with pandas