Where to Begin

Feature Engineering with PySpark

John Hogue

Lead Data Scientist, General Mills

Diving Straight to Analysis

Here be Monsters

Monsters

Data Science Process

Check your versions!

  # return spark version
  spark.version

  # return python version
  import sys
  sys.version_info

Data is supplied as Parquet

Stored Column-wise
- Fast to query column subsets
Structured, defined schema
- Fields and Data Types defined
- Great for messy text data
Industry Adopted
- Good skill to have! ?

Parquet File Format

PySpark read methods

# JSON
spark.read.json('example.json')

# CSV or delimited files
spark.read.csv('example.csv')

# Parquet
spark.read.parquet('example.parq')

# Read a parquet file to a PySpark DataFrame
df = spark.read.parquet('example.parq')

Feature Engineering with PySpark