词袋表示法

Python 中的自然语言处理（NLP）

Fouad Trad

Machine Learning Engineer

NLP 流程回顾

完整流程图，指出第 3 和第 4 章聚焦 transformers 库。

词袋模型（BoW）

将文本表示为数字的基础技术
通过统计每个词出现次数来表示文本
把词放进"袋子"并计数
忽略语法和顺序

图片展示将文本词语投入一个袋子，然后统计各词出现次数。

BoW 示例

图片显示两句：'I love NLP' 和 'I love machine learning.'

BoW 示例

图片显示由句子得到的词表：I、love、NLP、machine、learning。

构建所有唯一词的词表

BoW 示例

图片显示按已定义词表计数得到的每句特征向量。

构建所有唯一词的词表
统计词表中每个词的出现次数

BoW 实现代码

reviews = ["I loved the movie. It was amazing!",
           "The movie was okay.",
           "I hated the movie. It was boring."]

def preprocess(text):

    text = text.lower()

    tokens = word_tokenize(text)

    tokens = [word for word in tokens if word not in string.punctuation]

    return " ".join(tokens)

cleaned_reviews = [preprocess(review) for review in reviews]

print(cleaned_reviews)

['i loved the movie it was amazing', 
 'the movie was okay', 
 'i hated the movie it was boring']

BoW 实现代码

from sklearn.feature_extraction.text import CountVectorizer


vectorizer = CountVectorizer()

vectorizer.fit(cleaned_reviews)

print(vectorizer.get_feature_names_out())

['amazing' 'boring' 'hated' 'it' 'loved' 'movie' 'okay' 'the' 'was']

BoW 输出

X = vectorizer.transform(cleaned_reviews)

# OR
X = vectorizer.fit_transform(cleaned_reviews)

print(X)

<Compressed Sparse Row sparse matrix of dtype 'int64'
    with 16 stored elements and shape (3, 9)>

稀疏矩阵：大部分元素为 0 的表

BoW 输出

print(X.toarray())

[[1 0 0 1 1 1 0 1 1]
 [0 0 0 0 0 1 1 1 1]
 [0 1 1 1 0 1 0 1 1]]

print(vectorizer.get_feature_names_out())

['amazing' 'boring' 'hated' 'it' 'loved' 'movie' 'okay' 'the' 'was']

词频

import numpy as np


word_counts = np.sum(X.toarray(), axis=0)

words = vectorizer.get_feature_names_out()

import matplotlib.pyplot as plt


plt.bar(words, word_counts)

plt.title("Word Frequencies in Movie Reviews")

plt.xlabel("Words")
plt.ylabel("Frequency")
plt.show()

柱状图显示词及其频率，停用词最常见。

让我们来练习！

Python 中的自然语言处理（NLP）