构建词袋模型

Python 中的 NLP 特征工程

Rounak Banik

Data Scientist

ML 算法的数据格式回顾

对于任意 ML 算法，

数据必须为表格形式
训练特征必须为数值型

词袋模型

提取词元
计算词元频率
用这些频率和语料库词表构建词向量

词袋模型示例

语料

"The lion is the king of the jungle"

"Lions have lifespans of a decade"

"The lion is an endangered species"

词袋模型示例

词表 → a, an, decade, endangered, have, is, jungle, king, lifespans, lion, Lions, of, species, the, The

"The lion is the king of the jungle"

[0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 2, 1]

"Lions have lifespans of a decade"

[1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0]

"The lion is an endangered species"

[0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1]

文本预处理

Lions、lion → lion
The、the → the
无标点
无停用词
词表更小
降维可提升性能

使用 sklearn 的词袋模型

corpus = pd.Series([
    'The lion is the king of the jungle',
    'Lions have lifespans of a decade',
    'The lion is an endangered species'
])

使用 sklearn 的词袋模型

# Import CountVectorizer
from sklearn.feature_extraction.text import CountVectorizer

# Create CountVectorizer object
vectorizer = CountVectorizer()

# Generate matrix of word vectors
bow_matrix = vectorizer.fit_transform(corpus)
print(bow_matrix.toarray())

array([[0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 3],
       [0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0],
       [1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1]], dtype=int64)

让我们练习！

Python 中的 NLP 特征工程