Permutation tests

Foundations of Inference in Python

Paul Savala

Assistant Professor of Mathematics

Permutation tests

Two groups of small figures, set apart from one another, each with different colors.

Permutation tests

Shuffles samples
Observes outcome
Observed difference looks like a random outcome?

Small figures being swapped for one another.

Permutation tests in SciPy

new_satisfaction = [94, 85, 79, 91, 82]
old_satisfcation = [90, 87, 77, 85, 82]


# Group together our data
data = (new_satisfaction, old_satisfcation)


# Define our test statistic
def statistic(x, y):
    return np.mean(x) - np.mean(y)


# Compute a permutation test for the difference in means
stats.permutation_test(data, statistic, n_resamples=1000,
                       vectorized=False, alternative='greater')

Permutation tests for correlation

A line graph with a red and blue line. The lines are slightly correlated.

stats.pearsonr(red_data, blue_data)[0]

0.08

Permutation tests for correlation

data = (red_data, blue_data)


def statistic(x, y):
    return stats.pearsonr(x, y)[0]


res = stats.permutation_test(data, statistic, n_resamples=1000,
                             vectorized=False, alternative='two-sided')


print(res.pvalue < 0.05)

False

Permutation tests

Build a null distribution by randomly shuffling data
Tests for significance of an outcome

Bootstrapping

Build a probability distribution by randomly sampling data
Creates a confidence interval showing most likely outcomes

Let's practice!

Foundations of Inference in Python