Cross-validation

Supervised Learning with scikit-learn

George Boorman

Core Curriculum Manager, DataCamp

Cross-validation motivation

Model performance is dependent on the way we split up the data
Not representative of the model's ability to generalize to unseen data
Solution: Cross-validation!

Cross-validation basics

table headings: split 1, fold 1, fold 2, fold 3, fold 4, and fold 5

Cross-validation basics

split 1 reserved as a test set

Cross-validation basics

folds 2-5 used as training data

Cross-validation basics

compute metric on these folds

Cross-validation basics

Fold 2 as test data

Cross-validation basics

folds 1, 3, 4, and 5 as training data

Cross-validation basics

calculate metric again

Cross-validation basics

repeat with the third fold

Cross-validation basics

repeat with fourth fold

Cross-validation basics

repeat with the fifth fold

Cross-validation and model performance

5 folds = 5-fold CV
10 folds = 10-fold CV
k folds = k-fold CV
More folds = More computationally expensive

Cross-validation in scikit-learn

from sklearn.model_selection import cross_val_score, KFold

kf = KFold(n_splits=6, shuffle=True, random_state=42)

reg = LinearRegression()

cv_results = cross_val_score(reg, X, y, cv=kf)

Evaluating cross-validation peformance

print(cv_results)

[0.70262578, 0.7659624, 0.75188205, 0.76914482, 0.72551151, 0.73608277]

print(np.mean(cv_results), np.std(cv_results))

0.7418682216666667 0.023330243960652888

print(np.quantile(cv_results, [0.025, 0.975]))

array([0.7054865, 0.76874702])

Let's practice!

Supervised Learning with scikit-learn