GAM to learn non-linear transformations

Supervised Learning di R: Regresi

Nina Zumel and John Mount

Win-Vector, LLC

Generalized Additive Models (GAMs)

$$ y \sim b0 + s1(x1) + s2(x2) + .... $$

Supervised Learning di R: Regresi

Learning Non-linear Relationships

Supervised Learning di R: Regresi

gam() in the mgcv package

gam(formula, family, data)

family:

  • gaussian (default): "regular" regression
  • binomial: probabilities
  • poisson/quasipoisson: counts

Best for larger datasets

Supervised Learning di R: Regresi

The s() function

anx ~ s(hassles)
  • s() designates that variable should be non-linear
  • Use s() with continuous variables
    • More than about 10 unique values
Supervised Learning di R: Regresi

Revisit the hassles data

Supervised Learning di R: Regresi

Revisit the hassles data

Model RMSE (cross-val) $R^2$ (training)
Linear ($hassles$) 7.69 0.53
Quadratic ($hassles^2$) 6.89 0.63
Cubic ($hassles^3$) 6.70 0.65
Supervised Learning di R: Regresi

GAM of the hassles data

model <- gam(
  anx ~ s(hassles), 
  data = hassleframe, 
  family = gaussian
)

summary(model)

...

R-sq.(adj) =  0.619   Deviance explained = 64.1%
GCV = 49.132  Scale est. = 45.153    n = 40
Supervised Learning di R: Regresi

Examining the Transformations

plot(model)

$y$ values: predict(model, type = "terms")

Supervised Learning di R: Regresi

Predicting with the Model

predict(model, newdata = hassleframe, type = "response")

Supervised Learning di R: Regresi

Comparing out-of-sample performance

Knowing the correct transformation is best, but GAM is useful when transformation isn't known

Model RMSE (cross-val) $R^2$ (training)
Linear ($hassles$) 7.69 0.53
Quadratic ($hassles^2$) 6.89 0.63
Cubic ($hassles^3$) 6.70 0.65
GAM 7.06 0.64
  • Small dataset $\rightarrow$ noisier GAM
Supervised Learning di R: Regresi

Let's practice!

Supervised Learning di R: Regresi

Preparing Video For Download...