Adaptive resampling

Hyperparameter Tuning in R

Dr. Shirin Elsinghorst

Senior Data Scientist

What is Adaptive Resampling?

Grid Search

All hyperparameter combinations are computed.

Random Search

Random subsets of hyperparameter combinations are computed.

→ Evaluation of best combination is done at the end.

Adaptive Resampling

Hyperparameter combinations are resampled with values near combinations that performed well.
Adaptive Resampling is, therefore, faster and more efficient!

"Futility Analysis in the Cross-Validation of Machine Learning Models." Max Kuhn; ARXIV 2014

Adaptive resampling in caret

trainControl: method = "adaptive_cv" + search = "random" + adaptive =

min: minimum number of resamples per hyperparameter
alpha: confidence level for removing hyperparameters
method: "gls" for linear model or "BT" for Bradley-Terry
complete: if TRUE generates full resampling set

fitControl <- trainControl(method = "adaptive_cv",
                             adaptive = list(min = 2, alpha = 0.05, 
                                             method = "gls", complete = TRUE),
                             search = "random")

trainControl() + tuneLength = x

fitControl <- trainControl(method = "adaptive_cv", number = 3, repeats = 3,
                           adaptive = list(min = 2, 
                                           alpha = 0.05, 
                                           method = "gls", 
                                           complete = TRUE),
                             search = "random")

tic()
set.seed(42)
gbm_model_voters_adaptive <- train(turnout16_2016 ~ ., 
                                   data = voters_train_data, 
                                   method = "gbm", 
                                   trControl = fitControl,
                                   verbose = FALSE,
                                   tuneLength = 7)
toc()

1239.837 sec elapsed

Adaptive resampling

gbm_model_voters_adaptive

...
Resampling results across tuning parameters:
  shrinkage   interaction.depth  n.minobsinnode  n.trees  Accuracy   Kappa       Resamples
  0.07137493   5                  6              4152     0.9564654  0.02856571  9        
  0.08408739   5                 14               674     0.9547185  0.02098853  4        
  0.28552325   8                 15              3209     0.9568141  0.03024238  3        
  0.33663932  10                 13              2595     0.9571130  0.04250979  9        
  0.54251480   3                 24              3683     0.9482171  0.03568586  2        
  0.56406870   7                 25              4685     0.9549898  0.05284333  5        
  0.58695763   8                 24              1431     0.9520286  0.02742592  2        
Accuracy was used to select the optimal model using the largest value.
The final values used for the model were n.trees = 2595,
interaction.depth = 10, shrinkage = 0.3366393 and n.minobsinnode = 13.

Let's get coding!

Hyperparameter Tuning in R