Log

Merancang Pipeline Peramalan untuk Produksi

Rami Krispin

Senior Manager, Data Science and Engineering

Log pipeline

Pencatatan (Logging)

Metadata
Uji validasi
Status
Kinerja

Log pipeline

Pencatatan (Logging)

Metadata
Uji validasi
Status
Kinerja

Fungsi

Observabilitas
Debugging
Keputusan berbasis peran

Menyiapkan log

Tetapkan skema log
Tangani skenario:
- Tidak ada data baru
- Hasil validasi
Tambahkan log

Proses log ETL

Periksa Tugas API

import etl.eia_etl as ee

log = ee.Log()

facet = {'respondent': 'US48', 
         'type': 'D'}

log.create_log(facets = facets)

Skema Log

Proses log ETL

Periksa Tugas API

import etl.eia_etl as ee

log = ee.Log()

facet = {'respondent': 'US48', 
         'type': 'D'}

log.create_log(facets = facets)

Skema Log

Proses log ETL

Periksa Tugas API

import etl.eia_etl as ee

log = ee.Log()

facet = {'respondent': 'US48', 
         'type': 'D'}

log.create_log(facets = facets)

Skema Log

Proses log ETL

Periksa Tugas API

import etl.eia_etl as ee

log = ee.Log()

facet = {'respondent': 'US48', 
         'type': 'D'}

log.create_log(facets = facets)

Skema Log

Proses log ETL

Periksa Tugas API

import etl.eia_etl as ee

log = ee.Log()

facet = {'respondent': 'US48', 
         'type': 'D'}

log.create_log(facets = facets)

Skema Log

Proses log ETL

Periksa Tugas API

import etl.eia_etl as ee

log = ee.Log()

facet = {'respondent': 'US48', 
         'type': 'D'}

log.create_log(facets = facets)

Skema Log

Proses log ETL

Periksa Tugas API

import etl.eia_etl as ee

log = ee.Log()

facet = {'respondent': 'US48', 
         'type': 'D'}

log.create_log(facets = facets)

Skema Log

Proses log ETL

Status API

meta = ee.get_metadata(api_key = api_key,
api_path = api_meta_path, 
meta_path= log_path, 
facets = facets, 
offset = 22, window = 336)

Status API

Proses log ETL

Status Data

meta = ee.get_metadata(api_key = api_key,
api_path = api_meta_path, 
meta_path= log_path, 
facets = facets, 
offset = 22, window = 336)

Status API

Proses log ETL

Tugas Validasi

if not meta.updates_available:
  log.no_updates()

Status API

Proses log ETL

Tugas Validasi

Status API

Proses log ETL

Tugas Penyegaran Data

get = ea.eia_get(api_key=api_key, 
    api_path= api_data_path, 
    data = "value", facets = facets, 
    start = meta.start, 
    end = meta.api_end_offset)

Keluaran Data

Proses log ETL

Gagal Menyegarkan Data

log.failure()

Proses log ETL

Tugas Validasi Data

test = ee.Validation(data = get.data,  
tbl_name= "get request", 
label = "validation",
parameters=get.parameters,initial= False,
warning=0.10, error=0, critical=0)

schema_refresh = pb.Schema(
    columns=[
        ("index", "datetime64[ns]"),   
        ("respondent", "object"),
        ("respondent-name", "object"),
        ("type", "object"),
        ("type-name", "object"),
        ("value", "int64"),
        ("value-units", "object")
    ])

test.add_schema(schema = schema_refresh)

Proses log ETL

Tugas Validasi Data

Ringkasan Validasi Data

Proses log ETL

Kegagalan Validasi

log.failure()

Proses log ETL

Tambahkan Data

if log.log["status"]:
  print("Appending the data")
  df = ee.AppendData()
  df.append_data(data_path = data_path, 
    new_data = get.data,save = True,
    schema = schema_append, 
    parameters = get.parameters)
  print(df.validation)
  if df.status and df.save:
    log.log["update"] = True
  else:
    log.log["update"] = False