Charger du texte en langage naturel

Introduction à Spark SQL en Python

Mark Plutowski

Data Scientist

Le jeu de données

The Project Gutenberg eBook of The Adventures of Sherlock Holmes,

par Sir Arthur Conan Doyle.

Disponible sur gutenberg.org

Introduction à Spark SQL en Python

Charger du texte

df = spark.read.text('sherlock.txt')

print(df.first()) 

Row(value='The Project Gutenberg EBook of The Adventures of Sherlock Holmes')

print(df.count())

5500
Introduction à Spark SQL en Python

Charger du parquet

df1 = spark.read.load('sherlock.parquet')
Introduction à Spark SQL en Python

Texte chargé

df1.show(15, truncate=False)

+--------------------------------------------------------------------+
|value                                                               |
+--------------------------------------------------------------------+
|The Project Gutenberg EBook of The Adventures of Sherlock Holmes    |
|by Sir Arthur Conan Doyle                                           |
|(#15 in our series by Sir Arthur Conan Doyle)                       |
|                                                                    |
|Copyright laws are changing all over the world. Be sure to check the|
|copyright laws for your country before downloading or redistributing|
|this or any other Project Gutenberg eBook.                          |
|                                                                    |
|This header should be the first thing seen when viewing this Project|
|Gutenberg file.  Please do not remove it.  Do not change or edit the|
|header without written permission.                                  |
|                                                                    |
|Please read the "legal small print," and other information about the|
|eBook and Project Gutenberg at the bottom of this file.  Included is|
|important information about your specific rights and restrictions in|
+--------------------------------------------------------------------+
Introduction à Spark SQL en Python

Passer en minuscules

df = df1.select(lower(col('value')))

print(df.first())

Row(lower(value)=
    'the project gutenberg ebook of the adventures of sherlock holmes')

df.columns

['lower(value)']
Introduction à Spark SQL en Python

Opération d’alias

df = df1.select(lower(col('value')).alias('v'))

df.columns

['v']
Introduction à Spark SQL en Python

Remplacer du texte

df = df1.select(regexp_replace('value', 'Mr\.', 'Mr').alias('v'))

"Mr. Holmes." ==> "Mr Holmes."

df = df1.select(regexp_replace('value', 'don\'t', 'do not').alias('v'))

"don't know." ==> "do not know."

Introduction à Spark SQL en Python

Tokeniser le texte

df = df2.select(split('v', '[ ]').alias('words'))
df.show(truncate=False)
Introduction à Spark SQL en Python

Tokenisation – sortie

+--------------------------------------------------------------------------------------+
|words                                                                                  |
+--------------------------------------------------------------------------------------+
|[the, project, gutenberg, ebook, of, the, adventures, of, sherlock, holmes]           |
|[by, sir, arthur, conan, doyle]                                                       |
|[(#15, in, our, series, by, sir, arthur, conan, doyle)]                               |
|[]                                                                                    
.
.
.
|[please, read, the, "legal, small, print,", and, other, information, about, the]      |
.
.
.
|[**welcome, to, the, world, of, free, plain, vanilla, electronic, texts**]            |
+--------------------------------------------------------------------------------------+
Introduction à Spark SQL en Python

Les séparateurs sont supprimés

punctuation = "_|.\?\!\",\'\[\]\*()"
df3 = df2.select(split('v', '[ %s]' % punctuation).alias('words'))

df3.show(truncate=False)
Introduction à Spark SQL en Python

Les séparateurs sont supprimés – sortie

+---------------------------------------------------------------------------------------+
|words                                                                                   |
+---------------------------------------------------------------------------------------+
|[the, project, gutenberg, ebook, of, the, adventures, of, sherlock, holmes]            |
|[by, sir, arthur, conan, doyle]                                                        |
|[, #15, in, our, series, by, sir, arthur, conan, doyle, ]                              |
|[]                                                                                     .
.
.

|[please, read, the, , legal, small, print, , , and, other, information, about, the]    |
.
.
.
[, , welcome, to, the, world, of, free, plain, vanilla, electronic, texts, , ]         |
++---------------------------------------------------------------------------------------+
Introduction à Spark SQL en Python

Éclater un tableau (explode)

df4 = df3.select(explode('words').alias('word'))
df4.show()
Introduction à Spark SQL en Python

Éclater un tableau – sortie

+----------+
|      word|
+----------+
|       the|
|   project|
| gutenberg|
|     ebook|
|        of|
|       the|
|adventures|
|        of|
|  sherlock|
|    holmes|
|        by|
|       sir|
|    arthur|
|     conan|
|     doyle|
+----------+
Introduction à Spark SQL en Python

Explode augmente le nombre de lignes

print(df3.count())

5500

print(df4.count())

131404
Introduction à Spark SQL en Python

Supprimer les lignes vides

print(df.count())

131404

nonblank_df = df.where(length('word') > 0)

print(nonblank_df.count())

107320
Introduction à Spark SQL en Python

Ajouter une colonne id de ligne

df2 = df.select('word', monotonically_increasing_id().alias('id'))

df2.show()
Introduction à Spark SQL en Python

Ajouter une colonne id de ligne – sortie

+----------+---+
|      word| id|
+----------+---+
|       the|  0|
|   project|  1|
| gutenberg|  2|
|     ebook|  3|
|        of|  4|
|       the|  5|
|adventures|  6|
|        of|  7|
|  sherlock|  8|
|    holmes|  9|
|        by| 10|
|       sir| 11|
|    arthur| 12|
|     conan| 13|
|     doyle| 14|
|       #15| 15|
+----------+---+
Introduction à Spark SQL en Python

Partitionner les données

df2 = df.withColumn('title', when(df.id < 25000, 'Préface')
                             .when(df.id < 50000, 'Chapitre 1')
                             .when(df.id < 75000, 'Chapitre 2')
                             .otherwise('Chapitre 3'))

df2 = df2.withColumn('part', when(df2.id < 25000, 0)
                            .when(df2.id < 50000, 1)
                            .when(df2.id < 75000, 2)
                            .otherwise(3))
                            .show()
Introduction à Spark SQL en Python

Partitionner les données – sortie

+----------+---+------------+----+
|word      |id |title       |part|
+----------+---+------------+----+
|the       |0  |     Préface|0   |
|project   |1  |     Préface|0   |
|gutenberg |2  |     Préface|0   |
|ebook     |3  |     Préface|0   |
|of        |4  |     Préface|0   |
|the       |5  |     Préface|0   |
|adventures|6  |     Préface|0   |
|of        |7  |     Préface|0   |
|sherlock  |8  |     Préface|0   |
|holmes    |9  |     Préface|0   |
Introduction à Spark SQL en Python

Répartir selon une colonne

df2 = df.repartition(4, 'part')

print(df2.rdd.getNumPartitions())

4
Introduction à Spark SQL en Python

Lire du texte pré-partitionné

$ ls sherlock_parts

sherlock_part0.txt   
sherlock_part1.txt   
sherlock_part2.txt   
sherlock_part3.txt   
sherlock_part4.txt   
sherlock_part5.txt       
sherlock_part6.txt   
sherlock_part7.txt   
sherlock_part8.txt     
sherlock_part9.txt     
sherlock_part10.txt     
sherlock_part11.txt  
sherlock_part12.txt
sherlock_part13.txt
Introduction à Spark SQL en Python

Lire du texte pré-partitionné

df_parts = spark.read.text('sherlock_parts')
Introduction à Spark SQL en Python

Passons à la pratique !

Introduction à Spark SQL en Python

Preparing Video For Download...