Notazione a punti e SQL

Introduzione a Spark SQL in Python

Mark Plutowski

Data Scientist

La tabella ha 3 colonne

df.columns

['train_id', 'station', 'time']

df.show(5)

+--------+-------------+-----+
|train_id|      station| time|
+--------+-------------+-----+
|     324|San Francisco|7:59 |
|     324|  22nd Street|8:03 |
|     324|     Millbrae|8:16 |
|     324|    Hillsdale|8:24 |
|     324| Redwood City|8:31 |
+--------+-------------+-----+

Ci servono solo 2

df.select('train_id','station')
  .show(5)

+--------+-------------+
|train_id|      station|
+--------+-------------+
|     324|San Francisco|
|     324|  22nd Street|
|     324|     Millbrae|
|     324|    Hillsdale|
|     324| Redwood City|
+--------+-------------+

Tre modi per selezionare 2 colonne

df.select('train_id', 'station')
df.select(df.train_id, df.station)
from pyspark.sql.functions import col
df.select(col('train_id'), col('station'))

Due modi per rinominare una colonna

df.select('train_id','station')
  .withColumnRenamed('train_id','train')
  .show(5)

+-----+-------------+
|train|      station|
+-----+-------------+
|  324|San Francisco|
|  324|  22nd Street|
|  324|     Millbrae|
|  324|    Hillsdale|
|  324| Redwood City|
+-----+-------------+

df.select(col('train_id').alias('train'), 'station')

Non farlo!

df.select('train_id', df.station, col('time'))

Query SQL con notazione a punti

spark.sql('SELECT train_id AS train, station FROM schedule LIMIT 5')
     .show()

+-----+-------------+
|train|      station|
+-----+-------------+
|  324|San Francisco|
|  324|  22nd Street|
|  324|     Millbrae|
|  324|    Hillsdale|
|  324| Redwood City|
+-----+-------------+

df.select(col('train_id').alias('train'), 'station')
  .limit(5)
  .show()

SQL con funzioni finestra

query = """
SELECT *, 
ROW_NUMBER() OVER(PARTITION BY train_id ORDER BY time) AS id 
FROM schedule
"""
spark.sql(query)
     .show(11)

SQL con funzioni finestra

+--------+-------------+-----+---+
|train_id|      station| time| id|
+--------+-------------+-----+---+
|     217|       Gilroy|6:06 |  1|
|     217|   San Martin|6:15 |  2|
|     217|  Morgan Hill|6:21 |  3|
|     217| Blossom Hill|6:36 |  4|
|     217|      Capitol|6:42 |  5|
|     217|       Tamien|6:50 |  6|
|     217|     San Jose|6:59 |  7|
|     324|San Francisco|7:59 |  1|
|     324|  22nd Street|8:03 |  2|
|     324|     Millbrae|8:16 |  3|
|     324|    Hillsdale|8:24 |  4|
+--------+-------------+-----+---+

Funzione finestra con notazione a punti

from pyspark.sql import Window, 
from pyspark.sql.functions import row_number
df.withColumn("id", row_number()
                    .over(
                           Window.partitionBy('train_id')
                                 .orderBy('time')
                         )
  )

ROW_NUMBER in SQL: pyspark.sql.functions.row_number
L'interno della clausola OVER: pyspark.sql.Window
PARTITION BY: pyspark.sql.Window.partitionBy
ORDER BY: pyspark.sql.Window.orderBy

Uso di WindowSpec

La funzione over in Spark SQL corrisponde alla clausola OVER in SQL.
La classe pyspark.sql.window.Window rappresenta l'interno di una clausola OVER.

window = Window.partitionBy('train_id').orderBy('time')
dfx = df.withColumn('next', lead('time',1).over(window))

Sopra, type(window) è pyspark.sql.window.WindowSpec

Passiamo alla pratica !

Introduzione a Spark SQL in Python