Why run experiments?

A/B Testing in Python

Moe Lotfy, PhD

Principal Data Science Manager

The value of A/B testing

Reduce uncertainty around the impact of new designs and features
Decision-making --> scientific, evidence-based - not intuition
Generous value for the investment: simple changes lead to major wins
Continuous optimization at the mature stage of the business
Correlation does not imply causation

Hierarchy of evidence

Adapted version of hierarchy of evidence

¹ https://jamanetwork.com/journals/jama/article-abstract/392650

Do error messages reduce churn?

Microsoft Office 365 spurious correlation example:$^1$
- Spurious correlation: a strong correlation that appears to be causal but is not.

¹ Kohavi, Ron,Tang, Diane,Xu, Ya. Trustworthy Online Controlled Experiments. Cambridge University Press.

Pearson's correlation coefficient

A score that measures the strength of a linear relationship between two variables.
r>0: positive correlation
r = 0: neutral correlation
r<0: negative correlation

Pearson's correlation coefficient (r) formula:

The Pearson's correlation coefficient (r) equation

Assumes: Normal distribution and Linearity

Correlations visual inspection

# Import visualization library seaborn
import seaborn as sns

# Create pairplots
sns.pairplot(admissions[['Serial No.',\
      'GRE Score', 'Chance of Admit']])

Scatter plots showing the relationship between three columns in the dataset

Pearson correlation heatmap

# Import visualization library seaborn
import seaborn as sns

# Print Pearson correlation coefficient
print(admissions['GRE Score']\
   .corr(admissions['Chance of Admit']))

0.8026104595903503

# Plot correlations heatmap
sns.heatmap(admissions.corr(),annot=True)

Heatmap plot of all variables in the dataset and their correlation coefficients

Let's practice!

A/B Testing in Python