Evaluating hyperparameters with mlr

Hyperparameter Tuning in R

Dr. Shirin Elsinghorst

Data Scientist

Evaluation of our results can tell us:

How different hyperparameters affect the performance of our model.
Which hyperparameters have a particularly strong or weak impact on our model performance.
Whether our hyperparameter search converged, i.e. whether we can be reasonably confident that we found the most optimal hyperparameter combination (or close to it).

Recap

getParamSet("classif.h2o.deeplearning")

param_set <- makeParamSet(
  makeDiscreteParam("hidden", values = list(one = 10, two = c(10, 5, 10))),
  makeDiscreteParam("activation", values = c("Rectifier", "Tanh")),
  makeNumericParam("l1", lower = 0.0001, upper = 1),
  makeNumericParam("l2", lower = 0.0001, upper = 1)  )

ctrl_random <- makeTuneControlRandom(maxit = 50)

holdout <- makeResampleDesc("Holdout")

task <- makeClassifTask(data = knowledge_train_data, target = "UNS")
lrn <- makeLearner("classif.h2o.deeplearning", predict.type = "prob", 
                   fix.factors.prediction = TRUE)
lrn_tune <- tuneParams(lrn, task,
                       resampling = holdout,
                       control = ctrl_random,
                       par.set = param_set)

lrn_tune

generateHyperParsEffectData(lrn_tune, partial.dep = TRUE)

Tune result:
Op. pars: hidden=one; activation=Rectifier; l1=0.541; l2=0.229
mmce.test.mean=0.160000

HyperParsEffectData:
Hyperparameters: hidden,activation,l1,l2
Measures: mmce.test.mean
Optimizer: TuneControlRandom
Nested CV Used: FALSE
[1] "Partial dependence requested"
Snapshot of data:
  hidden activation         l1        l2 mmce.test.mean iteration exec.time
1    one  Rectifier 0.75940339 0.9956819           0.40         1     0.883
2    one  Rectifier 0.16701526 0.2948697           0.40         2     0.836
3    one  Rectifier 0.88458832 0.9228281           0.70         3     0.830
4    two  Rectifier 0.48840740 0.7276899           0.70         4     0.820
5    one       Tanh 0.87114452 0.9971268           0.40         5     0.835
6    two       Tanh 0.07412213 0.3841913           0.44         6     0.830

Plotting hyperparameter tuning results

hyperpar_effects <- generateHyperParsEffectData(lrn_tune, partial.dep = TRUE)
plotHyperParsEffect(hyperpar_effects, partial.dep.learn = "regr.randomForest",
                    x = "l1", y = "mmce.test.mean", z = "hidden",
                    plot.type = "line")

Now it's your turn!

Hyperparameter Tuning in R