Recursos de Tempo

Feature Engineering com PySpark

John Hogue

Lead Data Scientist, General Mills

A natureza cíclica das coisas

Imagem de tempo

Escolhendo o nível certo

Vendas diárias

Escolhendo o nível certo

Vendas mensais

Tratar campos de data como datas...

from pyspark.sql.functions import to_date

# Cast the data type to Date
df = df.withColumn('LISTDATE', to_date('LISTDATE'))

# Inspect the field
df[['LISTDATE']].show(2)

+----------+
|  LISTDATE|
+----------+
|2017-07-14|
|2017-10-08|
+----------+
only showing top 2 rows

Componentes de tempo

from pyspark.sql.functions import year, month

# Create a new column of year number
df = df.withColumn('LIST_YEAR', year('LISTDATE'))

# Create a new column of month number
df = df.withColumn('LIST_MONTH', month('LISTDATE'))

from pyspark.sql.functions import dayofmonth, weekofyear

# Create new columns of the day number within the month
df = df.withColumn('LIST_DAYOFMONTH', dayofmonth('LISTDATE'))

# Create new columns of the week number within the year
df = df.withColumn('LIST_WEEKOFYEAR', weekofyear('LISTDATE'))

Métricas básicas de tempo

Dias no mercado

from pyspark.sql.functions import datediff

# Calculate difference between two date fields
df.withColumn('DAYSONMARKET', datediff('OFFMARKETDATE', 'LISTDATE'))

Recursos defasados

Gota de café na xícara

window()

Retorna um registro com base em um grupo de registros

lag(col, count=1)

Retorna o valor deslocado por linhas antes da linha atual

Recursos defasados no PySpark

from pyspark.sql.functions import lag
from pyspark.sql.window import Window

# Create Window
w = Window().orderBy(m_df['DATE'])

# Create lagged column
m_df = m_df.withColumn('MORTGAGE-1wk', lag('MORTGAGE', count=1).over(w))

# Inspect results
m_df.show(3)

+----------+------------+----------------+
|      DATE|    MORTGAGE|    MORTGAGE-1wk|
+----------+------------+----------------+
|2013-10-10|        4.23|            null|
|2013-10-17|        4.28|            4.23|
|2013-10-24|        4.13|            4.28|
+----------+------------+----------------+
only showing top 3 rows

É HORA de praticar!

Feature Engineering com PySpark