Ingesti data

Merancang Pipeline Peramalan untuk Produksi

Rami Krispin

Senior Manager, Data Science and Engineering

Proses ETL

Proses ETL - sumber

Proses Ingesti Data

Proses ETL - ekstraksi

Proses Ingesti Data

Proses ETL - transformasi

Proses Ingesti Data

Proses ETL - pemuatan

Proses Ingesti Data

Proses ETL - validasi data

Proses Ingesti Data

Proses ETL - integritas data

Proses Ingesti Data

Proses ETL - pencatatan (logging)

Proses Ingesti Data

Proses ETL - penyegaran (refresh)

Proses Ingesti Data

Proses ETL - penyegaran (refresh)

Proses Ingesti Data

Proses ETL - penyegaran (refresh)

Proses Ingesti Data

Proses ETL - penyegaran (refresh)

Proses Ingesti Data

Proses ETL - memicu pipeline baru

Proses Ingesti Data

Proses ETL - pembaruan data ternormalisasi

Proses Ingesti Data

Proses ETL - pemeriksaan paritas data

Tugas Cek API

Proses ETL - metadata

Tugas Status Data

Proses ETL - penyegaran data

Tugas Penyegaran Data

Proses ETL - validasi data

Tugas Validasi Data

Proses ETL - analisis data

Tugas Analisis Validasi Data

Proses ETL - menambahkan data

Tugas Menambahkan Data

Proses ETL - penyegaran peramalan

Tugas Penyegaran Peramalan

Proses ETL - evaluasi peramalan

Tugas Penilaian Peramalan

Proses ETL - tanpa pembaruan

Tugas Validasi

Proses ETL - kegagalan penyegaran

Tugas Validasi

Proses ETL - kegagalan validasi

Tugas Validasi

Pemeriksaan validasi data

Skema data
Rentang nilai
Nilai hilang
Duplikasi
Logika bisnis

Pemeriksaan validasi data

print(raw)

period    respondent    respondent-name    type    type-name    value    value-units
0    2025-05-01 00:00:00    US48    United States Lower 48    D    Demand    504242    
1    2025-04-30 23:00:00    US48    United States Lower 48    D    Demand    508099    
2    2025-04-30 22:00:00    US48    United States Lower 48    D    Demand    508323    
3    2025-04-30 21:00:00    US48    United States Lower 48    D    Demand    500551    
4    2025-04-30 20:00:00    US48    United States Lower 48    D    Demand    492240

Pemeriksaan validasi data

import pointblank as pb

table_schema =  pb.Schema(
    columns=[
        ("index", "datetime64[ns]"),   
        ("respondent", "object"),
        ("respondent-name", "object"),
        ("type", "object"),
        ("type-name", "object"),
        ("value", "int64"),
        ("value-units", "object")
    ]
    )

Pemeriksaan validasi data

validation = (pb.Validate(data=raw,
              tbl_name="US48 Data Validation",
              label="Data Refresh",
              thresholds=pb.Thresholds(warning=0.2, error=0, critical=0.1))

.col_schema_match(schema=table_schema)

.col_vals_gt(columns="value", value=0)

.col_vals_in_set(columns="respondent", set = ["US48"])
.col_vals_in_set(columns="type", set = ["D"])

.col_vals_not_null(columns=["period", "value"])

.rows_distinct()

.interrogate())

Pemeriksaan validasi data

Validasi Data

Pemeriksaan validasi data

print(validation.all_passed())

True

Ayo berlatih!

Merancang Pipeline Peramalan untuk Produksi