Multiple testing

Practicing Statistics Interview Questions in Python

Conor Dewey

Data Scientist, Squarespace

Multiple comparisons problem

  Jelly bean example

1 xkcd
Practicing Statistics Interview Questions in Python

Correcting for multiple comparisons

Graph pad

1 GraphPad
Practicing Statistics Interview Questions in Python

Common approaches

  • Bonferroni correction
  • Sidak correction
  • Step-based procedures
  • Tukey’s procedure
  • Dunnet’s correction
Practicing Statistics Interview Questions in Python

Bonferroni correction

Bonferroni correction formula

1 Wikimedia
Practicing Statistics Interview Questions in Python

Example

from statsmodels.sandbox.stats.multicomp import multipletests
p_adjusted = multipletests(pvals, alpha=.05, method='bonferroni')
print(p_adjusted[0])
print(p_adjusted[1])

[ True False False False False]
[0.05 0.25 0.5  1.   1.  ]
Practicing Statistics Interview Questions in Python

Side effects

Side effects of testing

1 What’s wrong with Bonferroni adjustments
Practicing Statistics Interview Questions in Python

Summary

  • Multiple comparisons problem
  • Common correction approaches
  • Bonferroni correction
Practicing Statistics Interview Questions in Python

Let's prepare for the interview!

Practicing Statistics Interview Questions in Python

Preparing Video For Download...