Metode de ansamblu

Exersează întrebări de interviu pentru Machine Learning în Python

Lisa Stuart

Data Scientist

Tehnici de învățare prin ansamblu

Bootstrap Aggregation
Boosting
Stacking de modele

Error measurement

Short trees

Tall trees

Fat trees

Model liniar

Bias

Ipoteză de relație liniară (incorectă)

Bias ridicat
Subajustare
Generalizare slabă a modelului
Creșterea complexității reduce bias-ul

Model complex

Varianță

Modele cu complexitate ridicată:

Varianță ridicată
Supraajustare
Generalizare slabă a modelului

Compromisul Bias-Varianță

¹ Sursă: Elements of Statistical Learning de Trevor Hastie, Robert Tibshirani și Jerome Friedman

Bagging (Bootstrap aggregation)

Eșantioane bootstrap
- Subset selectat cu înlocuire
- Același rând poate fi ales de mai multe ori
Model construit pentru fiecare eșantion
Se face media rezultatelor
Reduce varianța

¹ https://medium.com/@rrfd/boosting-bagging-and-stacking-ensemble-methods-with-sklearn-and-mlens-a455c0c982de

Boosting

Mai multe modele construite secvențial
Predicțiile incorecte sunt ponderate
Reduce bias-ul

¹ https://blog.bigml.com/2017/03/14/introduction-to-boosted-trees/

Model stacking

Predicțiile Modelului 1
Predicțiile Modelului 2...
Predicțiile Modelului N
Stacking pentru modelul cu acuratețe maximă
- Folosește predicțiile modelului de bază (Modelul N) ca intrare pentru modelul de nivel 2

¹ http://supunsetunga.blogspot.com/

Vecstack package

# import modules
from sklearn.ensemble import BaggingClassifier
from sklearn.ensemble import AdaBoostClassifier
from xgboost import XGBClassifier
from vecstack import stacking

# Create list: stacked_models
stacked_models = [BaggingClassifier(n_estimators=25, random_state=123), AdaBoostClassifier(n_estimators=25, random_state=123)]

# Stack the models: stack_train, stack_test
stack_train, stack_test = stacking(stacked_models, X_train, y_train, X_test, regression=False, mode='oof_pred_bag', 
                                   needs_proba=False, metric=accuracy_score, n_folds=4, stratified=True, shuffle=True, random_state=0, verbose=2)

# Initialize and fit 2nd level model
final_model = XGBClassifier(random_state=123, n_jobs=-1, learning_rate=0.1, n_estimators=10, max_depth=3)
final_model_fit = final_model.fit(stack_train, y_train)

# Predict: stacked_pred
stacked_pred = final_model.predict(stack_test)

# Final prediction score
print('Final prediction score: [%.8f]' % accuracy_score(y_test, stacked_pred))

¹ https://towardsdatascience.com/automate-stacking-in-python-fc3e7834772e

Ensemble functions

Algorithm	Function
Bootstrap aggregation	`sklearn.ensemble.BaggingClassifier()`
Boosting	`sklearn.ensemble.AdaBoostClassifier()`
XGBoost	`xgboost.XGBClassifier()`

Bagging vs boosting

Technique	Bias	Variance
Bootstrap aggregation (Bagging)	Increase	Decrease
Boosting	Decrease	Increase

Major ensemble techniques MCQ

Which of the following statements is true about the three major techniques used for ensemble methods in Machine Learning? Select the statement that is true:

Boosting methods decrease model variance.
Boosting methods increase the predictive abilities of a classifier.
Bootstrap aggregation, or bagging, decreases model bias.
Model stacking takes the predictions from individual models and combines them to create a higher accuracy model.

Major ensemble techniques MCQ: answer

Which of the following statements is true about the three major techniques used for ensemble methods in Machine Learning? The correct answer is:

Model stacking takes the predictions from individual models and combines them to create a higher accuracy model. (The final model obtained from the predictions of several individual models almost always outperforms the individuals.)

Major ensemble techniques MCQ: incorrect answers

Which of the following statements is true about the three major techniques used for ensemble methods in Machine Learning?

Boosting methods decrease model variance. (Boosting methods decrease model bias which, at the same time, helps increase variance to find that sweet spot for best model generalization.)
Boosting methods increase the predictive abilities of a classifier. (Boosting decreases model bias, which may or may not increase the predictive abilities of a classifier.)
Bootstrap aggregation, or bagging, decreases model bias. (Bagging decreases model variance.)

Let's practice!

Exersează întrebări de interviu pentru Machine Learning în Python