Adaptive boosting: award winning model

Ensemble Methods in Python

Román de las Heras

Data Scientist, Appodeal

Award winning model

About AdaBoost:

  • Proposed by Yoav Freund and Robert Schapire (1997)
  • Winner of the Gödel Prize in (2003)
  • The first practical boosting algorithm
  • Highly used and well known ensemble method

godel-prize.jpg

Ensemble Methods in Python

AdaBoost properties

  1. Instances are drawn using a sample distribution
    • Difficult instances have higher weights
    • Initialized to be uniform
  2. Estimators are combined with a weighted majority voting
    • Good estimators are given higher weights
  3. Guaranteed to improve
  4. Classification and Regression

adaboost-sample.bmp

Ensemble Methods in Python

AdaBoost classifier with scikit-learn

AdaBoostClassifier

from sklearn.ensemble import AdaBoostClassifier

clf_ada = AdaBoostClassifier(
   base_estimator,
   n_estimators,
   learning_rate
)

Parameters

  • base_estimator
    • Default: Decision Tree (max_depth=1)
  • n_estimators
    • Default: 50
  • learning_rate
    • Default: 1.0
    • Trade-off between n_estimators and learning_rate
Ensemble Methods in Python

AdaBoost regressor with scikit-learn

AdaBoostRegressor

from sklearn.ensemble import AdaBoostRegressor

reg_ada = AdaBoostRegressor(
   base_estimator,
   n_estimators,
   learning_rate,
   loss
)

Parameters

  • base_estimator
    • Default: Decision Tree (max_depth=3)
  • loss
    • linear (default)
    • square
    • exponential
Ensemble Methods in Python

Let's practice!

Ensemble Methods in Python

Preparing Video For Download...