Featuregeneratie

Feature Engineering met PySpark

John Hogue

Lead Data Scientist

Waarom nieuwe features genereren?

Generatie

Vermenigvuldigen

Optellen

Verschillen

Delen

Feature Engineering met PySpark

Waarom nieuwe features genereren?

Lengte lineair model-plot

Breedte lineair model-plot

Feature Engineering met PySpark

Twee features combineren

Vermenigvuldiging

# Creating a new feature, area by multiplying
df = df.withColumn('TSQFT', (df['WIDTH'] * df['LENGTH']))

Oppervlakte lineair model-plot

Feature Engineering met PySpark

Andere manieren om twee features te combineren

# Sum two columns
df = df.withColumn('TSQFT', (df['SQFTBELOWGROUND'] + df['SQFTABOVEGROUND']))

# Divide two columns
df = df.withColumn('PRICEPERTSQFT', (df['LISTPRICE'] / df['TSQFT']))

# Difference two columns
df = df.withColumn('DAYSONMARKET', datediff('OFFMARKETDATE', 'LISTDATE'))
Feature Engineering met PySpark

Wat is de grens?

Automatisering van features

  • FeatureTools & TSFresh
  • Explosie aan features
  • Hogere orde en verder?

Futuristisch beeld

Feature Engineering met PySpark

Ga eropuit en combineer!

Feature Engineering met PySpark

Preparing Video For Download...