Introduction au codage du texte

Ingénierie des caractéristiques pour le Machine Learning en Python

Robert O'Callaghan

Director of Data Science, Ordergroove

Normaliser votre texte

Exemple de texte libre :

Citoyens et citoyennes du Sénat et de la Chambre des représentants : PARMI les vicissitudes de la vie, aucun événement n'aurait pu m'emplir d'une plus grande anxiété que celui dont l'avis a été transmis sur votre ordre et reçu le e jour du mois en cours.

Ensemble de données

print(speech_df.head())

                  Name           Inaugural Address    \ 
0    George Washington     First Inaugural Address
1    George Washington    Second Inaugural Address
2    John Adams                  Inaugural Address    
3    Thomas Jefferson      First Inaugural Address    
4    Thomas Jefferson     Second Inaugural Address

                        Date                               text
0    Thursday, April 30, 1789    Fellow-Citizens of the Sena...
1       Monday, March 4, 1793    Fellow Citizens: I AM again...
2     Saturday, March 4, 1797    WHEN it was first perceived...
3    Wednesday, March 4, 1801    Friends and Fellow-Citizens...
4       Monday, March 4, 1805    PROCEEDING, fellow-citizens...

Retirer les caractères indésirables

[a-zA-Z] : toutes les lettres
[^a-zA-Z] : tous les caractères non alphabétiques

speech_df['text'] = speech_df['text']\
                   .str.replace('[^a-zA-Z]', ' ')

Retirer les caractères indésirables

Avant :

"Fellow-Citizens of the Senate and of the House of  
Representatives: AMONG the vicissitudes incident to   
life no event could have filled me with greater" ...

Après :

"Fellow Citizens of the Senate and of the House of  
Representatives AMONG the vicissitudes incident to   
life no event could have filled me with greater" ...

Uniformiser la casse

speech_df['text'] = speech_df['text'].str.lower()
print(speech_df['text'][0])

"fellow citizens of the senate and of the house of  
representatives among the vicissitudes incident to   
life no event could have filled me with greater"...

Longueur du texte

speech_df['char_cnt'] = speech_df['text'].str.len()
print(speech_df['char_cnt'].head())

0    1889  
1     806  
2    2408  
3    1495  
4    2465
Name: char_cnt, dtype: int64

Comptage des mots

speech_df['word_cnt'] = 
    speech_df['text'].str.split()
speech_df['word_cnt'].head(1)

['fellow', 'citizens', 'of', 'the', 'senate', 'and',...

Comptage des mots

speech_df['word_counts'] = 
    speech_df['text'].str.split().str.len()
print(speech_df['word_splits'].head())

0    1432
1     135
2    2323
3    1736
4    2169
Name: word_cnt, dtype: int64

Longueur moyenne des mots

speech_df['avg_word_len'] = 
         speech_df['char_cnt'] / speech_df['word_cnt']

Passons à la pratique !

Ingénierie des caractéristiques pour le Machine Learning en Python