P-values, alpha, and errors

Experimental Design in Python

James Chapman

Curriculum Manager, DataCamp

P-values and alpha

 

  • P-values: probability of observing our data if the null hypothesis was true

  • $\alpha$: threshold at which we consider our results statistically significant

  • P-value $\le \alpha$: Reject null in favor of alternative hypothesis

 

Backgammon board

Experimental Design in Python

The dataset: crop yields

crop_yields.head()

  Fertilizer_Type  Crop_Yield
0         Organic      26.225
1       Synthetic      17.452
2         Organic      22.283
3       Synthetic      23.548
4       Synthetic      24.138
Experimental Design in Python

Visualizing the data

import seaborn as sns
sns.displot(data=crop_data, x="Crop_Yield", hue="Fertilizer_Type", kind="kde")

Kernal Density Estimator comparative plot

Experimental Design in Python

Conducting an independent samples t-test

$\alpha = 0.05$

from scipy.stats import ttest_ind
organic_yield = crop_yields[crop_yields['Fertilizer_Type'] == 'Organic']
['Crop_Yield']
synthetic_yield = crop_yields[crop_yields['Fertilizer_Type'] == 'Synthetic']
['Crop_Yield']
t_stat, p_val = ttest_ind(organic_yield, synthetic_yield)
print(p_val)

1.8496748715743899e-209
Experimental Design in Python

Exploring experimental errors

A table summarizing statistical errors: Type I errors occur when we incorrectly reject a true null hypothesis. Type II errors happen when we fail to reject a false null hypothesis.

  • Type I Error: False positive, rejecting a true null hypothesis
  • Type II Error: False negative, failing to reject a false null hypothesis
Experimental Design in Python

More on alpha

  • Common values: 0.05, 0.01, and 0.10
    • Reflecting a 5%, 1%, and 10% probability of making a Type I error
  • Choosing an $\alpha$
    • Based on the context of the study
    • Balancing a tolerance for a Type I error
  • Conventions:
    • 0.05 (5%): Most common, used as a convention
    • 0.01 (1%): More stringest testing, where cost of Type I error is high
    • 0.10 (10%): Sometimes in preliminary studies, where higher tolerance for Type I error
Experimental Design in Python

Let's practice!

Experimental Design in Python

Preparing Video For Download...