文本编码简介
Python 中的机器学习特征工程
Robert O'Callaghan
Director of Data Science, Ordergroove
标准化文本
自由文本示例：
参议院和众议院的各位公民：在人生变迁中，没有哪件事比你们依命通知并于本月某日收到的那件事更令我忧虑。
数据集
print(speech_df.head())
Name Inaugural Address \
0 George Washington First Inaugural Address
1 George Washington Second Inaugural Address
2 John Adams Inaugural Address
3 Thomas Jefferson First Inaugural Address
4 Thomas Jefferson Second Inaugural Address
Date text
0 Thursday, April 30, 1789 Fellow-Citizens of the Sena...
1 Monday, March 4, 1793 Fellow Citizens: I AM again...
2 Saturday, March 4, 1797 WHEN it was first perceived...
3 Wednesday, March 4, 1801 Friends and Fellow-Citizens...
4 Monday, March 4, 1805 PROCEEDING, fellow-citizens...
移除不需要的字符
[a-zA-Z]：所有字母字符[^a-zA-Z]：所有非字母字符
speech_df['text'] = speech_df['text']\
.str.replace('[^a-zA-Z]', ' ')
移除不需要的字符
之前：
"Fellow-Citizens of the Senate and of the House of
Representatives: AMONG the vicissitudes incident to
life no event could have filled me with greater" ...
之后：
"Fellow Citizens of the Senate and of the House of
Representatives AMONG the vicissitudes incident to
life no event could have filled me with greater" ...
统一大小写
speech_df['text'] = speech_df['text'].str.lower()
print(speech_df['text'][0])
"fellow citizens of the senate and of the house of
representatives among the vicissitudes incident to
life no event could have filled me with greater"...
文本长度
speech_df['char_cnt'] = speech_df['text'].str.len()
print(speech_df['char_cnt'].head())
0 1889
1 806
2 2408
3 1495
4 2465
Name: char_cnt, dtype: int64
词计数
speech_df['word_cnt'] =
speech_df['text'].str.split()
speech_df['word_cnt'].head(1)
['fellow', 'citizens', 'of', 'the', 'senate', 'and',...
词计数
speech_df['word_counts'] =
speech_df['text'].str.split().str.len()
print(speech_df['word_splits'].head())
0 1432
1 135
2 2323
3 1736
4 2169
Name: word_cnt, dtype: int64
平均词长
speech_df['avg_word_len'] =
speech_df['char_cnt'] / speech_df['word_cnt']
让我们来练习！
Python 中的机器学习特征工程
