調整 CART 的超參數

Machine Learning with Tree-Based Models in Python

Elie Kawerk

Data Scientist

超參數

機器學習模型：

  • 參數：從資料中學得

    • CART 範例：節點的分割點、節點的分割特徵等

  • 超參數：非由資料學得，訓練前設定

    • CART 範例：max_depth、min_samples_leaf、分割準則等
Machine Learning with Tree-Based Models in Python

What is hyperparameter tuning?

  • 問題：為學習演算法搜尋一組最佳超參數。

  • 解法：找出能產生最佳模型的超參數組合。

  • 最佳模型：帶來最佳的分數。

  • 分數：在 sklearn 預設為 accuracy（分類）與 $R^2$（迴歸）。

  • 使用交叉驗證估計泛化效能。

Machine Learning with Tree-Based Models in Python

Why tune hyperparameters?

  • In sklearn, a model's default hyperparameters are not optimal for all problems.

  • Hyperparameters should be tuned to obtain the best model performance.

Machine Learning with Tree-Based Models in Python

超參數調校的方法

  • Grid Search

  • Random Search

  • Bayesian Optimization

  • Genetic Algorithms

  • ....

Machine Learning with Tree-Based Models in Python

網格搜尋交叉驗證

  • 手動設定離散的超參數網格。

  • 設定用來評分模型表現的指標。

  • 在網格上做窮舉搜尋。

  • 對每組超參數，評估模型的 CV 分數。

  • 最佳超參數為取得最佳 CV 分數之模型的那一組。

Machine Learning with Tree-Based Models in Python

網格搜尋交叉驗證：範例

  • 超參數網格：
    • max_depth = {2,3,4},
    • min_samples_leaf = {0.05, 0.1}
  • 超參數空間 = { (2,0.05) , (2,0.1) , (3,0.05), ... }
  • CV 分數 = { $score_{(2,0.05)}$ , ... }
  • 最佳超參數 = 對應最佳 CV 分數的那組超參數。
Machine Learning with Tree-Based Models in Python

檢視 sklearn 中 CART 的超參數

# Import DecisionTreeClassifier
from sklearn.tree import DecisionTreeClassifier

# Set seed to 1 for reproducibility
SEED = 1

# Instantiate a DecisionTreeClassifier 'dt'
dt = DecisionTreeClassifier(random_state=SEED)
Machine Learning with Tree-Based Models in Python

檢視 sklearn 中 CART 的超參數

# Print out 'dt's hyperparameters
print(dt.get_params())

        {'class_weight': None,
         'criterion': 'gini',
         'max_depth': None,
         'max_features': None,
         'max_leaf_nodes': None,
         'min_impurity_decrease': 0.0,
         'min_impurity_split': None,
         'min_samples_leaf': 1,
         'min_samples_split': 2,
         'min_weight_fraction_leaf': 0.0,
         'presort': False,
         'random_state': 1,
         'splitter': 'best'}
Machine Learning with Tree-Based Models in Python
# Import GridSearchCV
from sklearn.model_selection import GridSearchCV

# Define the grid of hyperparameters 'params_dt'
params_dt = {
             'max_depth': [3, 4,5, 6],
             'min_samples_leaf': [0.04, 0.06, 0.08],
             'max_features': [0.2, 0.4,0.6, 0.8]
            }

# Instantiate a 10-fold CV grid search object 'grid_dt'
grid_dt = GridSearchCV(estimator=dt, 
                       param_grid=params_dt,
                       scoring='accuracy',                       
                       cv=10,
                       n_jobs=-1)

# Fit 'grid_dt' to the training data
grid_dt.fit(X_train, y_train)
Machine Learning with Tree-Based Models in Python

擷取最佳超參數

# Extract best hyperparameters from 'grid_dt'
best_hyperparams = grid_dt.best_params_
print('Best hyerparameters:\n', best_hyperparams)

Best hyerparameters:
  {'max_depth': 3, 'max_features': 0.4, 'min_samples_leaf': 0.06}

# Extract best CV score from 'grid_dt'
best_CV_score = grid_dt.best_score_
print('Best CV accuracy'.format(best_CV_score))

Best CV accuracy: 0.938
Machine Learning with Tree-Based Models in Python

擷取最佳估計器

# Extract best model from 'grid_dt'
best_model = grid_dt.best_estimator_


# Evaluate test set accuracy
test_acc = best_model.score(X_test,y_test)

# Print test set accuracy
print("Test set accuracy of best model: {:.3f}".format(test_acc))

Test set accuracy of best model: 0.947
Machine Learning with Tree-Based Models in Python

一起來練習吧！

Machine Learning with Tree-Based Models in Python

Preparing Video For Download...