Backtesting

Merancang Pipeline Peramalan untuk Produksi

Rami Krispin

Senior Manager, Data Science and Engineering

Backtesting

Backtesting: membagi data menjadi beberapa partisi (window) untuk melatih dan menguji model

Backtesting

Split pelatihan

Backtesting

Split pengujian

Backtesting

Pendekatan expanding window menambah ukuran sampel tiap partisi

Backtesting

Pendekatan sliding window menjaga ukuran sampel tiap partisi tetap

Backtesting

Pada kedua pendekatan, ukuran test set tetap sama

Backtesting

Boxplot skor RMSE untuk lima model: Lasso, LightGBM, Linear Regression, Ridge, dan XGBoost

Pengaturan backtesting

Model

XGBoost
Light Gradient-Boosting Machine
Ridge
Lasso
Regresi Linear

Pengaturan backtesting

Pengaturan Partisi

Parameter backtesting

Pengaturan backtesting

Pengaturan Partisi

Parameter backtesting

Pengaturan backtesting

Pengaturan Partisi

Parameter backtesting

Pengaturan backtesting

Pengaturan Partisi

Parameter backtesting

Pengaturan backtesting

Ketidakpastian model

Interval prediksi konformal
- Dapat menangani model nonparametrik & parametrik
Tingkat signifikansi 95%

Library yang dibutuhkan

from mlforecast import MLForecast
from mlforecast.target_transforms import Differences
from mlforecast.utils import PredictionIntervals
from window_ops.expanding import expanding_mean
from lightgbm import LGBMRegressor
from xgboost import XGBRegressor
from sklearn.linear_model import Lasso, LinearRegression, Ridge
from sklearn.neural_network import MLPRegressor
from sklearn.ensemble import RandomForestRegressor
from utilsforecast.plotting import plot_series

Library yang dibutuhkan

import pandas as pd
import numpy as np
import requests
import json
import os
import datetime
from statistics import mean

Persiapan data

ts = pd.read_csv("data/data.csv")
ts["ds"] = pd.to_datetime(ts["ds"])
ts = ts.sort_values("ds")
ts = ts[["unique_id", "ds", "y"]]


end = ts["ds"].max()
start = end - datetime.timedelta(hours = 24 * 31 * 25)
ts = ts[ts["ds"] >= start]


os.environ['NIXTLA_ID_AS_COL'] = '1'

Definisikan model peramalan

ml_models = {
    "lightGBM": LGBMRegressor(n_estimators=500, verbosity=-1),
    "xgboost": XGBRegressor(),
    "linear_regression": LinearRegression(),
    "lasso": Lasso(),
    "ridge": Ridge()
}


mlf = MLForecast(
    models= ml_models,  
    freq='h', 
    lags=list(range(1, 24)),  
    date_features=["month", "day", "dayofweek", "week", "hour"])

Set parameter backtesting

Pengaturan Window

partitions = 10  
step_size = 24  
h = 72

Pengaturan Interval Prediksi

n_windows = 5
method = "conformal_distribution"
pi = PredictionIntervals(h=h, n_windows = n_windows , method = method)
levels = [95]

Melatih model dengan backtesting

bkt_df = mlf.cross_validation(
        df = ts,
        h = h,
        step_size = step_size,
        n_windows = partitions,
        prediction_intervals = pi, 
        level = levels)

Melatih model dengan backtesting

print(bkt_df.head())

     unique_id    ds                     cutoff                 y            lightGBM         
0    1            2024-04-22 00:00:00    2024-04-21 23:00:00    421082.60    421089.155837    
1    1            2024-04-22 01:00:00    2024-04-21 23:00:00    429728.30    425700.453391    
2    1            2024-04-22 02:00:00    2024-04-21 23:00:00    430690.96    424382.613668    
3    1            2024-04-22 03:00:00    2024-04-21 23:00:00    420094.58    409967.877157    
4    1            2024-04-22 04:00:00    2024-04-21 23:00:00    403292.36    393175.446116

Hasil backtesting format lebar

Informasi tentang DataFrame bkt_df, berisi 19 kolom termasuk unique_id, ds, y, dan nilai model

Ayo berlatih!

Merancang Pipeline Peramalan untuk Produksi