Feature-Erzeugung

Feature Engineering mit PySpark

John Hogue

Lead Data Scientist

Warum neue Features erzeugen?

Erzeugung

Multiplizieren

Addieren

Differenz bilden

Dividieren

Längen-Linear-Modell-Plot

Breiten-Linear-Modell-Plot

Multiplikation

# Creating a new feature, area by multiplying
df = df.withColumn('TSQFT', (df['WIDTH'] * df['LENGTH']))

Flächen-Linear-Modell-Plot

# Sum two columns
df = df.withColumn('TSQFT', (df['SQFTBELOWGROUND'] + df['SQFTABOVEGROUND']))

# Divide two columns
df = df.withColumn('PRICEPERTSQFT', (df['LISTPRICE'] / df['TSQFT']))

# Difference two columns
df = df.withColumn('DAYSONMARKET', datediff('OFFMARKETDATE', 'LISTDATE'))

Automatisierte Features

Futuristisches Bild

Feature Engineering mit PySpark