Een BoW-Naive Bayes-classifier bouwen
Feature Engineering voor NLP in Python
Rounak Banik
Data Scientist
Spamfiltering
| message | label |
|---|---|
| WINNER!! As a valued network customer you have been selected to receive a $900 prize reward! To claim call 09061701461 | spam |
| Ah, work. I vaguely remember that. What does it feel like? | ham |
Stappen
- Tekstvoorbewerking
- Een bag-of-words-model (representatie) bouwen
- Machine learning
Tekstvoorbewerking met CountVectorizer
CountVectorizer-argumenten
lowercase:False,Truestrip_accents:'unciode','ascii',Nonestop_words:'english',list,Nonetoken_pattern:regextokenizer:function
Het BoW-model bouwen
# Import CountVectorizer from sklearn.feature_extraction.text import CountVectorizer# Maak een CountVectorizer-object vectorizer = CountVectorizer(strip_accents='ascii', stop_words='english', lowercase=False)# Import train_test_split from sklearn.model_selection import train_test_split # Splits in train- en testsets X_train, X_test, y_train, y_test = train_test_split(df['message'], df['label'], test_size=0.25)
Het BoW-model bouwen
... ... # Genereer BoW-trainvectors X_train_bow = vectorizer.fit_transform(X_train)# Genereer BoW-testvectors X_test_bow = vectorizer.transform(X_test)
De Naive Bayes-classifier trainen
# Import MultinomialNB from sklearn.naive_bayes import MultinomialNB# Maak een MultinomialNB-object clf = MultinomialNB()# Train clf clf.fit(X_train_bow, y_train)# Bereken accuraatheid op de testset accuracy = clf.score(X_test_bow, y_test) print(accuracy)
0.760051
Laten we oefenen!
Feature Engineering voor NLP in Python
