使用 mlr 评估超参数

R 中的超参数调优

Dr. Shirin Elsinghorst

Data Scientist

结果评估可以告诉我们：

不同超参数如何影响模型性能。
哪些超参数对性能的影响特别强或弱。
超参数搜索是否收敛，即我们是否可较有把握找到最优的超参数组合（或接近）。

回顾

getParamSet("classif.h2o.deeplearning")

param_set <- makeParamSet(
  makeDiscreteParam("hidden", values = list(one = 10, two = c(10, 5, 10))),
  makeDiscreteParam("activation", values = c("Rectifier", "Tanh")),
  makeNumericParam("l1", lower = 0.0001, upper = 1),
  makeNumericParam("l2", lower = 0.0001, upper = 1)  )

ctrl_random <- makeTuneControlRandom(maxit = 50)

holdout <- makeResampleDesc("Holdout")

task <- makeClassifTask(data = knowledge_train_data, target = "UNS")
lrn <- makeLearner("classif.h2o.deeplearning", predict.type = "prob", 
                   fix.factors.prediction = TRUE)
lrn_tune <- tuneParams(lrn, task,
                       resampling = holdout,
                       control = ctrl_random,
                       par.set = param_set)

lrn_tune

generateHyperParsEffectData(lrn_tune, partial.dep = TRUE)

调参结果：
最优参数：hidden=one; activation=Rectifier; l1=0.541; l2=0.229
mmce.test.mean=0.160000

HyperParsEffectData：
超参数：hidden,activation,l1,l2
度量：mmce.test.mean
优化器：TuneControlRandom
使用嵌套CV：FALSE
[1] "请求了部分依赖"
数据快照：
  hidden activation         l1        l2 mmce.test.mean iteration exec.time
1    one  Rectifier 0.75940339 0.9956819           0.40         1     0.883
2    one  Rectifier 0.16701526 0.2948697           0.40         2     0.836
3    one  Rectifier 0.88458832 0.9228281           0.70         3     0.830
4    two  Rectifier 0.48840740 0.7276899           0.70         4     0.820
5    one       Tanh 0.87114452 0.9971268           0.40         5     0.835
6    two       Tanh 0.07412213 0.3841913           0.44         6     0.830

可视化超参数调参结果

hyperpar_effects <- generateHyperParsEffectData(lrn_tune, partial.dep = TRUE)
plotHyperParsEffect(hyperpar_effects, partial.dep.learn = "regr.randomForest",
                    x = "l1", y = "mmce.test.mean", z = "hidden",
                    plot.type = "line")

轮到你了！

R 中的超参数调优