Pipeline architecture

Concevoir des pipelines de prévision pour la production

Rami Krispin

Senior Manager, Data Science and Engineering

Model deployment

Experimentation Process

Model deployment

ETL and ML Pipelines

Pipelines requirements

Data ingestion

Refresh frequency - daily

Forecast refresh

Refresh frequency - daily
Forecast horizon - 72 hours

Robust

Unit testing and validation steps
Logs
Easy to maintain

Pipeline design

Pipeline requirements, including API requests, data transformation, forecast refresh, and logging

Pipeline design

Data ingestion process

Pipeline design

Forecasting automation

Pipeline design

Data storage

Pipeline design

Logging

Pipeline design

Tools used in pipeline - Airflow, mlflow, nixtla

Model registry

Approaches

Requirements

MLflow flavor
Customized function
Fitted object
Predicted method

Model registry

from lightgbm import LGBMRegressor
from mlforecast import MLForecast
import mlflow
import mlforecast.flavor


experiment_name = "ml_forecast"
mlflow_path = "file:///mlruns"


meta = mlflow.get_experiment_by_name(experiment_name)

Model registry

model = LGBMRegressor(n_estimators = 500, learning_rate= 0.05)


params = {
  "freq": "h",
  "lags": list(range(1, 24)),
  "date_features": ["month", "day", "dayofweek", "week", "hour"] 
}

Model registry

mlf = MLForecast(
    models= model,  
    freq= params["freq"], 
    lags=params["lags"],
    date_features=params["date_features"]
)

mlf.fit(ts)

Model registry

run_time = datetime.datetime.now().strftime("%Y-%m-%d %H-%M-%S")
run_name = f"lightGBM6_{run_time}"


print(run_name)

'lightGBM6_2025-05-19 05-12-16'

with mlflow.start_run(experiment_id=meta.experiment_id, 
                      run_name=run_name) as run:
  mlforecast.flavor.log_model(model=mlf, artifact_path="prod_model")

Model registry

Model registry with run name highlighted

Model registry

Model registry with model name highlighted

Model registry

Model registry metadata

Let's practice!

Concevoir des pipelines de prévision pour la production