Addressing complexities in experimental data

Experimental Design in Python

James Chapman

Curriculum Manager, DataCamp

Geological data

mineral_rocks
 SampleID     RockType  Location  MineralHardness  RockPorosity
        1  Metamorphic      West              5.9          12.3
        2      Igneous     North              5.3           1.6
        3  Metamorphic      East              5.6          11.0
        4  Metamorphic     South              3.2          12.2
        5  Sedimentary     South              2.0          29.8
Experimental Design in Python

Understanding data complexities

 

  • Rock types and mineral hardness may interact, impacting mineral properties
  • Rock porosity variance may vary, indicating heteroscedasticity
  • Confounding variables could influence mineral hardness and porosity

A collection of different rocks in a mound.

Experimental Design in Python

Addressing interactions

sns.scatterplot(x='MineralHardness', y='RockPorosity', 
                hue='RockType', data=mineral_rocks)

Mineral Hardness vs. Rock Porosity by Rock Type

Experimental Design in Python

Addressing heteroscedasticity

  • Heteroscedasticity: changing variability of a variable across the range of another variable
sns.residplot(x='MineralHardness', y='RockPorosity', 
              data=mineral_rocks, lowess=True)

Examining unequal variance

Experimental Design in Python

Non-normal data

sns.displot(mineral_rocks['RockPorosity'])

RockPorosity histogram and density curve

Experimental Design in Python

Data transformation with Box-Cox

from scipy.stats import boxcox
mineral_rocks['TransformedRockPorosity'], _ = boxcox(mineral_rocks['RockPorosity'])
sns.displot(mineral_rocks['TransformedRockPorosity'])

Box-Cox transformed Porosity

Experimental Design in Python
sns.residplot(x='MineralHardness', y='TransformedRockPorosity',
              data=mineral_rocks, lowess=True)

Residual plot of Transformed Rock Porosity against Mineral Hardness

Experimental Design in Python

Let's practice!

Experimental Design in Python

Preparing Video For Download...