AdaBoost

Machine Learning with Tree-Based Models in Python

Elie Kawerk

Data Scientist

Boosting

  • Boosting：將多個弱學習器組成一個強學習器的集成方法。

  • 弱學習器：表現僅略優於隨機猜測的模型。

  • 弱學習器範例：決策樁（最大深度為 1 的 CART）。

Machine Learning with Tree-Based Models in Python

Boosting

  • 依序訓練一組預測器。

  • 每個預測器試著修正前一個的錯誤。

  • 最常見的 Boosting 方法：

    • AdaBoost，

    • Gradient Boosting。

Machine Learning with Tree-Based Models in Python

Adaboost

  • 為 Adaptive Boosting 的縮寫。

  • 每個預測器會更重視前一個預測錯誤的樣本。

  • 透過調整訓練樣本權重達成。

  • 每個預測器會分配一個係數 $\alpha$。

  • $\alpha$ 取決於該預測器的訓練錯誤。

Machine Learning with Tree-Based Models in Python

AdaBoost：訓練

AdaBoost 訓練流程

Machine Learning with Tree-Based Models in Python

學習率（Learning Rate）

學習率：$0 < \eta \leq 1$ 學習率對 AdaBoost 的影響

Machine Learning with Tree-Based Models in Python

AdaBoost：預測

  • 分類：

    • 加權多數決。
    • 在 sklearn 中：AdaBoostClassifier。

  • 迴歸：

    • 加權平均。
    • 在 sklearn 中：AdaBoostRegressor。
Machine Learning with Tree-Based Models in Python

在 sklearn 中進行 AdaBoost 分類（Breast Cancer 資料集）

# Import models and utility functions
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

# Set seed for reproducibility
SEED = 1

# Split data into 70% train and 30% test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
                                                    stratify=y,
                                                    random_state=SEED)
Machine Learning with Tree-Based Models in Python
# Instantiate a classification-tree 'dt'
dt = DecisionTreeClassifier(max_depth=1, random_state=SEED)


# Instantiate an AdaBoost classifier 'adab_clf'
adb_clf = AdaBoostClassifier(base_estimator=dt, n_estimators=100)



# Fit 'adb_clf' to the training set
adb_clf.fit(X_train, y_train)

# Predict the test set probabilities of positive class
y_pred_proba = adb_clf.predict_proba(X_test)[:,1]



# Evaluate test-set roc_auc_score
adb_clf_roc_auc_score = roc_auc_score(y_test, y_pred_proba)
Machine Learning with Tree-Based Models in Python

在 sklearn 中進行 AdaBoost 分類（Breast Cancer 資料集）

# Print adb_clf_roc_auc_score
print('ROC AUC score: {:.2f}'.format(adb_clf_roc_auc_score)) 

ROC AUC score: 0.99
Machine Learning with Tree-Based Models in Python

一起來練習吧！

Machine Learning with Tree-Based Models in Python

Preparing Video For Download...