ウィンドウ関数 SQL

Pythonで学ぶ Spark SQL 入門

Mark Plutowski

Data Scientist

ウィンドウ関数 SQL とは

ドット記法や複雑なクエリより簡潔に表現できる
各行は他の行の値を用いて自分の値を計算する

列車時刻表

train_id	station	time
324	San Francisco	7:59
324	22nd Street	8:03
324	Millbrae	8:16
324	Hillsdale	8:24
324	Redwood City	8:31
324	Palo Alto	8:37
324	San Jose	9:05

次の停車までの時間列を追加

train_id	station	time	time_to_next_stop
324	San Francisco	7:59	4 分
324	22nd Street	8:03	13 分
324	Millbrae	8:16	8 分
324	Hillsdale	8:24	7 分
324	Redwood City	8:31	6 分
324	Palo Alto	8:37	28 分
324	San Jose	9:05	null

次の停車時刻の列

train_id	station	time	time (following row)
324	San Francisco	7:59	8:03
324	22nd Street	8:03	8:16
324	Millbrae	8:16	8:24
324	Hillsdale	8:24	8:31
324	Redwood City	8:31	8:37
324	Palo Alto	8:37	9:05
324	San Jose	9:05	null

OVER 句と ORDER BY 句

query = """
SELECT train_id, station, time, 
LEAD(time, 1) OVER (ORDER BY time) AS time_next 
FROM sched 
WHERE train_id=324 """

spark.sql(query).show()

+--------+-------------+-----+---------+
|train_id|      station| time|time_next|
+--------+-------------+-----+---------+
|     324|San Francisco|7:59 |    8:03 |
|     324|  22nd Street|8:03 |    8:16 |
|     324|     Millbrae|8:16 |    8:24 |
|     324|    Hillsdale|8:24 |    8:31 |
|     324| Redwood City|8:31 |    8:37 |
|     324|    Palo Alto|8:37 |    9:05 |
|     324|     San Jose|9:05 |    null |
+--------+-------------+-----+---------+

PARTITION BY 句

SELECT 
train_id, 
station, 
time, 
LEAD(time,1) OVER (PARTITION BY train_id ORDER BY time) AS time_next 
FROM sched

PARTITION BY 句を追加した結果

+--------+-------------+-----+---------+
|train_id|      station| time|time_next|
+--------+-------------+-----+---------+
|     217|       Gilroy|6:06 |    6:15 |
|     217|   San Martin|6:15 |    6:21 |
|     217|  Morgan Hill|6:21 |    6:36 |
|     217| Blossom Hill|6:36 |    6:42 |
|     217|      Capitol|6:42 |    6:50 |
|     217|       Tamien|6:50 |    6:59 |
|     217|     San Jose|6:59 |    null |
|     324|San Francisco|7:59 |    8:03 |
|     324|  22nd Street|8:03 |    8:16 |
|     324|     Millbrae|8:16 |    8:24 |
|     324|    Hillsdale|8:24 |    8:31 |
|     324| Redwood City|8:31 |    8:37 |
|     324|    Palo Alto|8:37 |    9:05 |
|     324|     San Jose|9:05 |    null |
+--------+-------------+-----+---------+

train_id	station	time	time_to_next_stop
324	San Francisco	7:59	4 分
324	22nd Street	8:03	13 分
324	Millbrae	8:16	8 分
324	Hillsdale	8:24	7 分
324	Redwood City	8:31	6 分
324	Palo Alto	8:37	28 分
324	San Jose	9:05	null

練習しましょう！

Pythonで学ぶ Spark SQL 入門