Model evaluation and implementation

Credit Risk Modeling in Python

Michael Crabtree

Data Scientist, Ford Motor Company

Comparing classification reports

Create the reports with classification_report() and compare

Classification report for logistic regression and gradient boosted tree

Formula for F1 score and macro average F1 score

ROC and AUC analysis

Models with better performance will have more lift
More lift means the AUC score is higher

ROC chart example with two models

Model calibration

We want our probabilities of default to accurately represent the model's confidence level
- The probability of default has a degree of uncertainty in it's predictions
A sample of loans and their predicted probabilities of default should be close to the percentage of defaults in that sample

Sample of loans	Average predicted PD	Sample percentage of actual defaults	Calibrated?
10	0.12	0.12	Yes
10	0.25	0.65	No

¹ http://datascienceassn.org/sites/default/files/Predicting%20good%20probabilities%20with%20supervised%20learning.pdf

Calculating calibration

Shows percentage of true defaults for each predicted probability
Essentially a line plot of the results of calibration_curve()

from sklearn.calibration import calibration_curve
calibration_curve(y_test, probabilities_of_default, n_bins = 5)

# Fraction of positives
(array([0.09602649, 0.19521012, 0.62035996, 0.67361111]),
# Average probability
 array([0.09543535, 0.29196742, 0.46898465, 0.65512207]))

Plotting calibration curves

plt.plot(mean_predicted_value, fraction_of_positives, label="%s" % "Example Model")

Calibration curve example

Checking calibration curves

As an example, two events selected (above and below perfect line)

Calibration curve with two examples of uncalibrated predictions

Calibration curve interpretation

Calibration curve with above example

Calibration curve interpretation

Calibration curve with below example

Let's practice!

Credit Risk Modeling in Python