Transformasi data dengan pandas

ETL dan ELT di Python

Jake Roach

Data Engineer

Mentranformasi data dalam pipeline

Data harus ditransformasi dengan tepat agar bernilai bagi pengguna berikutnya

Alur kerja transformasi data.

pandas menyediakan alat kuat untuk mentransformasi data tabel

.loc[]
.to_datetime()

Menyaring record dengan .loc[]

.loc[] memungkinkan kedua dimensi DataFrame ditransformasi

# Keep only non-zero entries
cleaned = raw_stock_data.loc[raw_stock_data["open"] > 0, :]

# Remove excess columns
cleaned = raw_stock_data.loc[:, ["timestamps", "open", "close"]]

# Combine into one step
cleaned = raw_stock_data.loc[raw_stock_data["open"] > 0, ["timestamps", "open", "close"]]

.iloc[] menggunakan pengindeksan bilangan bulat untuk memfilter DataFrame

cleaned = raw_stock_data.iloc[[0:50], [0, 1, 2]]

Mengubah tipe data

Tipe data sering perlu diubah untuk kebutuhan tahap lanjut

.to_datetime()

# "timestamps" column currectly looks like: "20230101085731"
# Convert "timestamps" column to type datetime
cleaned["timestamps"] = pd.to_datetime(cleaned["timestamps"], format="%Y%m%d%H%M%S")

Timestamp('2023-01-01 08:57:31')

# "timestamps" column currently looks like: 1681596000011
# Convert "timestamps" column to type datatime
cleaned["timestamps"] = pd.to_datetime(cleaned["timestamps"], unit="ms")

Timestamp('2023-04-15 22:00:00.011000')

Memvalidasi transformasi

Transformasi data memiliki risiko:

Kehilangan informasi
Membuat data cacat

# Several ways to investigate a DataFrame
cleaned = raw_stock_data.loc[raw_stock_data["open"] > 0, ["timestamps", "open", "close"]]
print(cleaned.head())

# Return smallest and largest records
print(cleaned.nsmallest(10, ["timestamps"]))
print(cleaned.nlargest(10, ["timestamps"]))

Ayo berlatih!

ETL dan ELT di Python