テキストエンコーディング入門

Python で学ぶ Machine Learning のための特徴量エンジニアリング

Robert O'Callaghan

Director of Data Science, Ordergroove

テキストの標準化

自由テキストの例:

Fellow-Citizens of the Senate and of the House of Representatives: AMONG the vicissitudes incident to life no event could have filled me with greater anxieties than that of which the notification was transmitted by your order, and received on the th day of the present month.

データセット

print(speech_df.head())

                  Name           Inaugural Address    \ 
0    George Washington     First Inaugural Address
1    George Washington    Second Inaugural Address
2    John Adams                  Inaugural Address    
3    Thomas Jefferson      First Inaugural Address    
4    Thomas Jefferson     Second Inaugural Address

                        Date                               text
0    Thursday, April 30, 1789    Fellow-Citizens of the Sena...
1       Monday, March 4, 1793    Fellow Citizens: I AM again...
2     Saturday, March 4, 1797    WHEN it was first perceived...
3    Wednesday, March 4, 1801    Friends and Fellow-Citizens...
4       Monday, March 4, 1805    PROCEEDING, fellow-citizens...

不要な文字の除去

[a-zA-Z]: すべての英字
[^a-zA-Z]: 英字以外すべて

speech_df['text'] = speech_df['text']\
                   .str.replace('[^a-zA-Z]', ' ')

不要な文字の除去

変更前:

"Fellow-Citizens of the Senate and of the House of  
Representatives: AMONG the vicissitudes incident to   
life no event could have filled me with greater" ...

変更後:

"Fellow Citizens of the Senate and of the House of  
Representatives AMONG the vicissitudes incident to   
life no event could have filled me with greater" ...

大文字小文字の統一

speech_df['text'] = speech_df['text'].str.lower()
print(speech_df['text'][0])

"fellow citizens of the senate and of the house of  
representatives among the vicissitudes incident to   
life no event could have filled me with greater"...

テキストの長さ

speech_df['char_cnt'] = speech_df['text'].str.len()
print(speech_df['char_cnt'].head())

0    1889  
1     806  
2    2408  
3    1495  
4    2465
Name: char_cnt, dtype: int64

単語数

speech_df['word_cnt'] = 
    speech_df['text'].str.split()
speech_df['word_cnt'].head(1)

['fellow', 'citizens', 'of', 'the', 'senate', 'and',...

単語数

speech_df['word_counts'] = 
    speech_df['text'].str.split().str.len()
print(speech_df['word_splits'].head())

0    1432
1     135
2    2323
3    1736
4    2169
Name: word_cnt, dtype: int64

平均単語長

speech_df['avg_word_len'] = 
         speech_df['char_cnt'] / speech_df['word_cnt']

Ayo berlatih!

Python で学ぶ Machine Learning のための特徴量エンジニアリング