Běžné slovní sekvence

Úvod do Spark SQL v Pythonu

Mark Plutowski

Data Scientist

Trénování

Úvod do Spark SQL v Pythonu

Predikce

Úvod do Spark SQL v Pythonu

Predikce koncového slova

Úvod do Spark SQL v Pythonu

Sekvence

Úvod do Spark SQL v Pythonu

Poslední prvek sekvence

Úvod do Spark SQL v Pythonu

Rychlá hnědá liška

Úvod do Spark SQL v Pythonu

Závorka věty

Úvod do Spark SQL v Pythonu

Jiný typ agregace

Úvod do Spark SQL v Pythonu

Videa

Úvod do Spark SQL v Pythonu

Kategorická data

Úvod do Spark SQL v Pythonu

Kategorická vs. ordinální data

  • Kategorická: he, hi, she, that, they
  • Ordinální: 1, 2, 3, 4, 5
Úvod do Spark SQL v Pythonu

Analýza sekvencí

Úvod do Spark SQL v Pythonu

Předcházející a následující slovo

Úvod do Spark SQL v Pythonu

3-tice

query3 = """
   SELECT 
   id,
   word AS w1,
   LEAD(word,1) OVER(PARTITION BY part ORDER BY id ) AS w2,
   LEAD(word,2) OVER(PARTITION BY part ORDER BY id ) AS w3
   FROM df
"""
Úvod do Spark SQL v Pythonu

SQL s okenní funkcí jako poddotaz

query3agg = """
SELECT w1, w2, w3, COUNT(*) as count FROM (
   SELECT 
   word AS w1,
   LEAD(word,1) OVER(PARTITION BY part ORDER BY id ) AS w2,
   LEAD(word,2) OVER(PARTITION BY part ORDER BY id ) AS w3
   FROM df
)
GROUP BY w1, w2, w3 
ORDER BY count DESC
""" 

spark.sql(query3agg).show()
Úvod do Spark SQL v Pythonu

SQL s okenní funkcí jako poddotaz – výstup

+-----+-----+-----+-----+
|   w1|   w2|   w3|count|
+-----+-----+-----+-----+
|  one|   of|  the|   49|
|    i|think| that|   46|
|   it|   is|    a|   46|
|   it|  was|    a|   45|
| that|   it|  was|   38|
|  out|   of|  the|   35|
|.....|.....|.....|.....|
Úvod do Spark SQL v Pythonu

Nejčastější 3-tice

+-----+-----+-----+-----+
|   w1|   w2|   w3|count|
+-----+-----+-----+-----+
|  one|   of|  the|   49|
|    i|think| that|   46|
|   it|   is|    a|   46|
|   it|  was|    a|   45|
| that|   it|  was|   38|
|  out|   of|  the|   35|
| that|    i| have|   35|
|there|  was|    a|   34|
|    i|   do|  not|   34|
| that|   it|   is|   33|
| that|   he|  was|   30|
| that|   he|  had|   30|
| that|    i|  was|   28|
+-----+-----+-----+-----+
Úvod do Spark SQL v Pythonu

Jiný typ agregace

query3agg = """
SELECT w1, w2, w3, length(w1)+length(w2)+length(w3) as length FROM (
   SELECT 
   word AS w1,
   LEAD(word,1) OVER(PARTITION BY part ORDER BY id ) AS w2,
   LEAD(word,2) OVER(PARTITION BY part ORDER BY id ) AS w3
   FROM df
   WHERE part <> 0 and part <> 13
)
GROUP BY w1, w2, w3 
ORDER BY length DESC
""" 

spark.sql(query3agg).show(truncate=False)
Úvod do Spark SQL v Pythonu

Jiný typ agregace

+-------------------+-------------------+---------------+------+
|                 w1|                 w2|             w3|length|
+-------------------+-------------------+---------------+------+
|comfortable-looking|           building|    two-storied|    38|
|         widespread|comfortable-looking|       building|    37|
|      extraordinary|      circumstances|      connected|    35|
|      simple-minded|      nonconformist|      clergyman|    35|
|       particularly|          malignant|  boot-slitting|    34|
|       unsystematic|        sensational|     literature|    33|
|       oppressively|        respectable|     frock-coat|    33|
|         relentless|        keen-witted|   ready-handed|    33|
|   travelling-cloak|                and|  close-fitting|    32|
|        ruddy-faced|      white-aproned|       landlord|    32|
|  fellow-countryman|            colonel|       lysander|    32|
+-------------------+-------------------+---------------+------+
Úvod do Spark SQL v Pythonu

Pojďme si procvičit

Úvod do Spark SQL v Pythonu

Preparing Video For Download...