Model Registration with MLflow

Concevoir des pipelines de prévision pour la production

Rami Krispin

Senior Manager, Data Science and Engineering

Launching the MLflow UI

mlflow ui

Terminal output displaying that the MLflow server has started from port 5000

Launching the MLflow UI

mlflow ui

Terminal output with the port http://127.0.0.1:5000 highlighted

http://127.0.0.1:5000

Analyze the backtesting results

MLflow UI

Analyze the backtesting results

MLflow UI - Experiments section highlighted, with options of Default and ml_forecast

Analyze the backtesting results

MLFlow UI with run names highlighted

Analyze the backtesting results

MLFlow UI with Group By option highlighted

Analyze the backtesting results

MLFlow UI with runs listed

Analyze the backtesting results

MLFlow UI with graphs displaying performance of each model by RMSE, MAPE, and Coverage

Analyze the backtesting results

MLFlow UI showing box plots with model RMSE score distributions

Can we improve the performance?

Model evaluation

Benchmark
Residuals analysis
Backtesting analysis

Potential improvements

Different models
New features
Tuning parameters

MLFlow UI showing box plots with model RMSE score distributions

Can we improve the performance?

Model optimization

Benchmark
Residuals analysis
Backtesting analysis

Potential improvements

Different models
New features
Tuning parameters

MLFlow UI showing box plots with model RMSE score distributions - LightGBM highlighted

Tuning parameters

lightGBM hyperparameters used

Tuning parameters

lightGBM hyperparameters used with learning_rate and n_estimates highlighted

Hypothesis

Using lower learning rate
Training with more trees

 ml_models2 = {
    "lightGBM1": LGBMRegressor(n_estimators = 100, learning_rate= 0.1),
    "lightGBM2": LGBMRegressor(n_estimators = 250, learning_rate= 0.1),
    "lightGBM3": LGBMRegressor(n_estimators = 500, learning_rate= 0.1),
    "lightGBM4": LGBMRegressor(n_estimators = 100, learning_rate= 0.05),
    "lightGBM5": LGBMRegressor(n_estimators = 250, learning_rate= 0.05),
    "lightGBM6": LGBMRegressor(n_estimators = 500, learning_rate= 0.05),
}

Analyzing the results

MLFlow UI showing performance of the models with different hyperparameters

Experimentation constraints

Experimentation and Deployment lifecycle covering train, test, evaluate, deploy, monitor, re-tune, and repeat

Let's practice!

Concevoir des pipelines de prévision pour la production