Where to Begin

Rekayasa Fitur dengan PySpark

John Hogue

Lead Data Scientist, General Mills

Diving Straight to Analysis

Here be Monsters

Monsters

Data Science Process

Check your versions!

  # return spark version
  spark.version

  # return python version
  import sys
  sys.version_info

Data is supplied as Parquet

Stored Column-wise
- Fast to query column subsets
Structured, defined schema
- Fields and Data Types defined
- Great for messy text data
Industry Adopted
- Good skill to have! 😃

Parquet File Format

PySpark read methods

# JSON
spark.read.json('example.json')

# CSV or delimited files
spark.read.csv('example.csv')

# Parquet
spark.read.parquet('example.parq')

# Read a parquet file to a PySpark DataFrame
df = spark.read.parquet('example.parq')

Rekayasa Fitur dengan PySpark