建立 bag-of-words 模型

Python 中文本特徵工程

Rounak Banik

Data Scientist

ML 演算法的資料格式回顧

適用於任何 ML 演算法，

資料需為表格形式
訓練特徵必須是數值型

Bag-of-words 模型

萃取字詞標記
計算字詞出現頻率
以頻率與語料庫字彙組成詞向量

Bag-of-words 範例

語料庫（Corpus）

"The lion is the king of the jungle"

"Lions have lifespans of a decade"

"The lion is an endangered species"

Bag-of-words 範例

字彙（Vocabulary） → a, an, decade, endangered, have, is, jungle, king, lifespans, lion, Lions, of, species, the, The

"The lion is the king of the jungle"

[0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 2, 1]

"Lions have lifespans of a decade"

[1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0]

"The lion is an endangered species"

[0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1]

文字前處理

Lions、lion → lion
The、the → the
去除標點
去除停用字
字彙更精簡
降低維度可提升效能

使用 sklearn 建立 bag-of-words 模型

corpus = pd.Series([
    'The lion is the king of the jungle',
    'Lions have lifespans of a decade',
    'The lion is an endangered species'
])

使用 sklearn 建立 bag-of-words 模型

# Import CountVectorizer
from sklearn.feature_extraction.text import CountVectorizer

# Create CountVectorizer object
vectorizer = CountVectorizer()

# Generate matrix of word vectors
bow_matrix = vectorizer.fit_transform(corpus)
print(bow_matrix.toarray())

array([[0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 3],
       [0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0],
       [1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1]], dtype=int64)

一起來練習吧！

Python 中文本特徵工程