Model Registration with MLflow

Designing Forecasting Pipelines for Production

Rami Krispin

Senior Manager, Data Science and Engineering

Launching the MLflow UI

mlflow ui

Terminal output displaying that the MLflow server has started from port 5000

Designing Forecasting Pipelines for Production

Launching the MLflow UI

mlflow ui

Terminal output with the port http://127.0.0.1:5000 highlighted

http://127.0.0.1:5000

Designing Forecasting Pipelines for Production

Analyze the backtesting results

MLflow UI

Designing Forecasting Pipelines for Production

Analyze the backtesting results

MLflow UI - Experiments section highlighted, with options of Default and ml_forecast

Designing Forecasting Pipelines for Production

Analyze the backtesting results

MLFlow UI with run names highlighted

Designing Forecasting Pipelines for Production

Analyze the backtesting results

MLFlow UI with Group By option highlighted

Designing Forecasting Pipelines for Production

Analyze the backtesting results

MLFlow UI with runs listed

Designing Forecasting Pipelines for Production

Analyze the backtesting results

MLFlow UI with graphs displaying performance of each model by RMSE, MAPE, and Coverage

Designing Forecasting Pipelines for Production

Analyze the backtesting results

MLFlow UI showing box plots with model RMSE score distributions

Designing Forecasting Pipelines for Production

Can we improve the performance?

Model evaluation

  • Benchmark
  • Residuals analysis
  • Backtesting analysis

 

Potential improvements

  • Different models
  • New features
  • Tuning parameters

MLFlow UI showing box plots with model RMSE score distributions

Designing Forecasting Pipelines for Production

Can we improve the performance?

Model optimization

  • Benchmark
  • Residuals analysis
  • Backtesting analysis

 

Potential improvements

  • Different models
  • New features
  • Tuning parameters

MLFlow UI showing box plots with model RMSE score distributions - LightGBM highlighted

Designing Forecasting Pipelines for Production

Tuning parameters

lightGBM hyperparameters used

Designing Forecasting Pipelines for Production

Tuning parameters

lightGBM hyperparameters used with learning_rate and n_estimates highlighted

Designing Forecasting Pipelines for Production

Hypothesis

  • Using lower learning rate
  • Training with more trees
 ml_models2 = {
    "lightGBM1": LGBMRegressor(n_estimators = 100, learning_rate= 0.1),
    "lightGBM2": LGBMRegressor(n_estimators = 250, learning_rate= 0.1),
    "lightGBM3": LGBMRegressor(n_estimators = 500, learning_rate= 0.1),
    "lightGBM4": LGBMRegressor(n_estimators = 100, learning_rate= 0.05),
    "lightGBM5": LGBMRegressor(n_estimators = 250, learning_rate= 0.05),
    "lightGBM6": LGBMRegressor(n_estimators = 500, learning_rate= 0.05),
}
Designing Forecasting Pipelines for Production

Analyzing the results

MLFlow UI showing performance of the models with different hyperparameters

Designing Forecasting Pipelines for Production

Experimentation constraints

Experimentation and Deployment lifecycle covering train, test, evaluate, deploy, monitor, re-tune, and repeat

Designing Forecasting Pipelines for Production

Let's practice!

Designing Forecasting Pipelines for Production

Preparing Video For Download...