Experimentation

Designing Forecasting Pipelines for Production

Rami Krispin

Senior Manager, Data Science and Engineering

Experimentation

Forecasting Pipeline

Designing Forecasting Pipelines for Production

Experimentation in a nutshell

Problem statement

Designing Forecasting Pipelines for Production

Experimentation in a nutshell

Setting the hypothesis

Designing Forecasting Pipelines for Production

Experimentation in a nutshell

Hypothesis testing

Designing Forecasting Pipelines for Production

Experimentation in a nutshell

Repeating the process for additional hypotheses

Designing Forecasting Pipelines for Production

Experimentation in a nutshell

Reaching a conclusion

Designing Forecasting Pipelines for Production

Experimentation in forecasting

Experimentation identifies the best (forecasting) modeling approach for each question.

  • Data
  • Models
  • Training framework
  • Performance KPIs
  • Model registration process
  • Model selection

A typical workflow covering data > modeling > model registration > model selection

Designing Forecasting Pipelines for Production

Experimentation in forecasting

DataFrame containing several forecasting models and their associated metrics - MAPE, RMSE, and Coverage

Designing Forecasting Pipelines for Production

Experimentation in forecasting

DataFrame with the AutoARIMA model highlighted

AutoARIMA documentation displaying several arguments that can be used when instantiating the model

Designing Forecasting Pipelines for Production

Experimentation in forecasting

DataFrame with the MSTL_ARIMA_trend model highlighted

MSTL_ARIMA_trend documentation highlighting that a different trend forecaster can be set

Designing Forecasting Pipelines for Production

Workflow

Workflow of train > test > evaluate > deploy > monitor > re-tune > repeat

Designing Forecasting Pipelines for Production

Workflow

Architecture consisting of data > data processing > backtesting > scoring > logging and metadata

Designing Forecasting Pipelines for Production

Workflow

Data processing

Designing Forecasting Pipelines for Production

Workflow

Backtesting

Designing Forecasting Pipelines for Production

Workflow

Scoring

Designing Forecasting Pipelines for Production

Workflow

Logging

Designing Forecasting Pipelines for Production

Workflow

Model settings defined in a JSON file

Designing Forecasting Pipelines for Production

Workflow

Technologies used in the architecture - pandas for data processing, nixtla for backtesting and scoring, and MLflow for logging

Designing Forecasting Pipelines for Production

Let's practice!

Designing Forecasting Pipelines for Production

Preparing Video For Download...