Monitoring Machine Learning in Python
Maciej Balawejder
Data Scientist
Estimated performance:
measures how well model is expected to perform
determined using estimators like CBPE, and DLE
Realized performance:
# Intialize the calculator
calc = nannyml.PerformanceCalculator(
y_pred_proba='y_pred_proba',
y_pred='y_pred',
y_true='arrived',
timestamp_column_name='timestamp',
problem_type='classification_binary',
chunk_period='d',
metrics=['roc_auc', 'accuracy'],
)
# Fit the calculator
calc.fit(reference)
realized_results = calc.calculate(analysis)
# Show realized performance plot
results.plot().show()
# Estimate and calculate results estimated_results = estimator.estimate(analysis) realized_results = calculator.calculate(analysis)
# Show comparison plot realized_results.compare(estimated_results).plot().show()
Monitoring Machine Learning in Python