Arsitektur pipeline

Merancang Pipeline Peramalan untuk Produksi

Rami Krispin

Senior Manager, Data Science and Engineering

Deploy model

Proses eksperimen

Deploy model

Pipeline ETL dan ML

Persyaratan pipeline

Ingesti data

Frekuensi penyegaran - harian

Penyegaran peramalan

Frekuensi penyegaran - harian
Horizon peramalan - 72 jam

Tangguh

Unit test dan langkah validasi
Log
Mudah dipelihara

Desain pipeline

Persyaratan pipeline, termasuk permintaan API, transformasi data, penyegaran peramalan, dan pencatatan log

Desain pipeline

Proses ingest data

Desain pipeline

Otomatisasi peramalan

Desain pipeline

Penyimpanan data

Desain pipeline

Pencatatan log

Desain pipeline

Alat pada pipeline - Airflow, mlflow, nixtla

Registri model

Pendekatan

Daftarkan semua model
Daftarkan hanya model terbaik

Persyaratan

Flavor MLflow
Fungsi kustom
Objek terlatih
Metode prediksi

Registri model

from lightgbm import LGBMRegressor
from mlforecast import MLForecast
import mlflow
import mlforecast.flavor


experiment_name = "ml_forecast"
mlflow_path = "file:///mlruns"


meta = mlflow.get_experiment_by_name(experiment_name)

Registri model

model = LGBMRegressor(n_estimators = 500, learning_rate= 0.05)


params = {
  "freq": "h",
  "lags": list(range(1, 24)),
  "date_features": ["month", "day", "dayofweek", "week", "hour"] 
}

Registri model

mlf = MLForecast(
    models= model,  
    freq= params["freq"], 
    lags=params["lags"],
    date_features=params["date_features"]
)

mlf.fit(ts)

Registri model

run_time = datetime.datetime.now().strftime("%Y-%m-%d %H-%M-%S")
run_name = f"lightGBM6_{run_time}"


print(run_name)

'lightGBM6_2025-05-19 05-12-16'

with mlflow.start_run(experiment_id=meta.experiment_id, 
                      run_name=run_name) as run:
  mlforecast.flavor.log_model(model=mlf, artifact_path="prod_model")

Registri model

Registri model dengan nama run disorot

Registri model

Registri model dengan nama model disorot

Registri model

Metadata registri model

Ayo berlatih!

Merancang Pipeline Peramalan untuk Produksi