詞袋（Bag-of-Words）表示法

Python 的 Natural Language Processing（NLP）

Fouad Trad

Machine Learning Engineer

NLP 工作流程回顧

完整工作流程圖，並註明第 3、4 章聚焦 transformers 函式庫。

詞袋模型（BoW）

將文字轉成數字的基礎方法
以每個詞出現的次數來表示文字
把詞丟進「袋子」裡並計數
忽略文法與順序

圖示：把文本的詞丟進袋子，再統計各詞出現次數。

BoW 範例

顯示兩個句子：'I love NLP' 與 'I love machine learning.'

BoW 範例

字彙來自句子的圖：I、love、NLP、machine、learning。

建立所有不重複詞的字彙表

BoW 範例

依字彙表計數後為每個句子產生特徵向量的圖示。

建立所有不重複詞的字彙表
計算字彙表中每個詞出現的次數

用程式實作 BoW

reviews = ["I loved the movie. It was amazing!",
           "The movie was okay.",
           "I hated the movie. It was boring."]

def preprocess(text):

    text = text.lower()

    tokens = word_tokenize(text)

    tokens = [word for word in tokens if word not in string.punctuation]

    return " ".join(tokens)

cleaned_reviews = [preprocess(review) for review in reviews]

print(cleaned_reviews)

['i loved the movie it was amazing', 
 'the movie was okay', 
 'i hated the movie it was boring']

用程式實作 BoW

from sklearn.feature_extraction.text import CountVectorizer


vectorizer = CountVectorizer()

vectorizer.fit(cleaned_reviews)

print(vectorizer.get_feature_names_out())

['amazing' 'boring' 'hated' 'it' 'loved' 'movie' 'okay' 'the' 'was']

BoW 輸出

X = vectorizer.transform(cleaned_reviews)

# OR
X = vectorizer.fit_transform(cleaned_reviews)

print(X)

<Compressed Sparse Row sparse matrix of dtype 'int64'
    with 16 stored elements and shape (3, 9)>

稀疏矩陣：大多數元素為 0 的表格

BoW 輸出

print(X.toarray())

[[1 0 0 1 1 1 0 1 1]
 [0 0 0 0 0 1 1 1 1]
 [0 1 1 1 0 1 0 1 1]]

print(vectorizer.get_feature_names_out())

['amazing' 'boring' 'hated' 'it' 'loved' 'movie' 'okay' 'the' 'was']

詞頻

import numpy as np


word_counts = np.sum(X.toarray(), axis=0)

words = vectorizer.get_feature_names_out()

import matplotlib.pyplot as plt


plt.bar(words, word_counts)

plt.title("Word Frequencies in Movie Reviews")

plt.xlabel("Words")
plt.ylabel("Frequency")
plt.show()

長條圖顯示詞與其頻率，停用詞最高。

一起來練習吧！

Python 的 Natural Language Processing（NLP）