建立 BoW 朴素貝葉斯分類器

Python 中文本特徵工程

Rounak Banik

Data Scientist

垃圾郵件過濾

message	label
WINNER!! As a valued network customer you have been selected to receive a $900 prize reward! To claim call 09061701461	垃圾信
Ah, work. I vaguely remember that. What does it feel like?	正常信

步驟

文字前處理
建立 bag-of-words（詞袋）表示
機器學習

用 CountVectorizer 做文字前處理

CountVectorizer 參數

lowercase: False、True
strip_accents: 'unciode'、'ascii'、None
stop_words: 'english'、list、None
token_pattern: regex
tokenizer: function

建立 BoW 模型

# Import CountVectorizer
from sklearn.feature_extraction.text import CountVectorizer


# Create CountVectorizer object
vectorizer = CountVectorizer(strip_accents='ascii', stop_words='english', lowercase=False)


# Import train_test_split
from sklearn.model_selection import train_test_split

# Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(df['message'], df['label'], test_size=0.25)

建立 BoW 模型

...
...
# Generate training Bow vectors
X_train_bow = vectorizer.fit_transform(X_train)


# Generate test BoW vectors
X_test_bow = vectorizer.transform(X_test)

訓練朴素貝葉斯分類器

# Import MultinomialNB
from sklearn.naive_bayes import MultinomialNB


# Create MultinomialNB object
clf = MultinomialNB()


# Train clf
clf.fit(X_train_bow, y_train)


# Compute accuracy on test set
accuracy = clf.score(X_test_bow, y_test)
print(accuracy)

0.760051

一起來練習吧！

Python 中文本特徵工程