Addressing complexities in experimental data

Experimental Design in Python

James Chapman

Curriculum Manager, DataCamp

Geological data

mineral_rocks

 SampleID     RockType  Location  MineralHardness  RockPorosity
        1  Metamorphic      West              5.9          12.3
        2      Igneous     North              5.3           1.6
        3  Metamorphic      East              5.6          11.0
        4  Metamorphic     South              3.2          12.2
        5  Sedimentary     South              2.0          29.8

Understanding data complexities

Rock types and mineral hardness may interact, impacting mineral properties
Rock porosity variance may vary, indicating heteroscedasticity
Confounding variables could influence mineral hardness and porosity

A collection of different rocks in a mound.

Addressing interactions

sns.scatterplot(x='MineralHardness', y='RockPorosity', 
                hue='RockType', data=mineral_rocks)

Mineral Hardness vs. Rock Porosity by Rock Type

Addressing heteroscedasticity

Heteroscedasticity: changing variability of a variable across the range of another variable

sns.residplot(x='MineralHardness', y='RockPorosity', 
              data=mineral_rocks, lowess=True)

Examining unequal variance

Non-normal data

sns.displot(mineral_rocks['RockPorosity'])

RockPorosity histogram and density curve

Data transformation with Box-Cox

from scipy.stats import boxcox
mineral_rocks['TransformedRockPorosity'], _ = boxcox(mineral_rocks['RockPorosity'])
sns.displot(mineral_rocks['TransformedRockPorosity'])

Box-Cox transformed Porosity

sns.residplot(x='MineralHardness', y='TransformedRockPorosity',
              data=mineral_rocks, lowess=True)

Residual plot of Transformed Rock Porosity against Mineral Hardness

Let's practice!

Experimental Design in Python