Introduction à l'encodage de texte

Feature engineering pour le Machine Learning en Python

Robert O'Callaghan

Director of Data Science, Ordergroove

Standardiser votre texte

Exemple de texte libre :

Citoyens du Sénat et de la Chambre des représentants : PARMI les vicissitudes de la vie, aucun événement n'aurait pu m'emplir d'une plus grande anxiété que celui dont la notification a été transmise sur votre ordre et reçue le e jour du mois en cours.

Feature engineering pour le Machine Learning en Python

Jeu de données

print(speech_df.head())

                  Name           Inaugural Address    \ 
0    George Washington     First Inaugural Address
1    George Washington    Second Inaugural Address
2    John Adams                  Inaugural Address    
3    Thomas Jefferson      First Inaugural Address    
4    Thomas Jefferson     Second Inaugural Address

                        Date                               text
0    Thursday, April 30, 1789    Fellow-Citizens of the Sena...
1       Monday, March 4, 1793    Fellow Citizens: I AM again...
2     Saturday, March 4, 1797    WHEN it was first perceived...
3    Wednesday, March 4, 1801    Friends and Fellow-Citizens...
4       Monday, March 4, 1805    PROCEEDING, fellow-citizens...
Feature engineering pour le Machine Learning en Python

Supprimer les caractères indésirables

  • [a-zA-Z] : Toutes les lettres
  • [^a-zA-Z] : Tous les caractères non alphabétiques
speech_df['text'] = speech_df['text']\
                   .str.replace('[^a-zA-Z]', ' ')
Feature engineering pour le Machine Learning en Python

Supprimer les caractères indésirables

Avant : 

"Fellow-Citizens of the Senate and of the House of  
Representatives: AMONG the vicissitudes incident to   
life no event could have filled me with greater" ...

Après : 

"Fellow Citizens of the Senate and of the House of  
Representatives AMONG the vicissitudes incident to   
life no event could have filled me with greater" ...
Feature engineering pour le Machine Learning en Python

Standardiser la casse

speech_df['text'] = speech_df['text'].str.lower()
print(speech_df['text'][0])

"fellow citizens of the senate and of the house of  
representatives among the vicissitudes incident to   
life no event could have filled me with greater"...
Feature engineering pour le Machine Learning en Python

Longueur du texte

speech_df['char_cnt'] = speech_df['text'].str.len()
print(speech_df['char_cnt'].head())

0    1889  
1     806  
2    2408  
3    1495  
4    2465
Name: char_cnt, dtype: int64
Feature engineering pour le Machine Learning en Python

Comptage des mots

speech_df['word_cnt'] = 
    speech_df['text'].str.split()
speech_df['word_cnt'].head(1)

['fellow', 'citizens', 'of', 'the', 'senate', 'and',...
Feature engineering pour le Machine Learning en Python

Comptage des mots

speech_df['word_counts'] = 
    speech_df['text'].str.split().str.len()
print(speech_df['word_splits'].head())

0    1432
1     135
2    2323
3    1736
4    2169
Name: word_cnt, dtype: int64
Feature engineering pour le Machine Learning en Python

Longueur moyenne des mots

speech_df['avg_word_len'] = 
         speech_df['char_cnt'] / speech_df['word_cnt']
Feature engineering pour le Machine Learning en Python

Passons à la pratique !

Feature engineering pour le Machine Learning en Python

Preparing Video For Download...