建立 n-gram 模型

Python 中文本特徵工程

Rounak Banik

Data Scientist

BoW 的侷限

review	label
`'The movie was good and not boring'`	positive
`'The movie was not good and boring'`	negative

n = 2，n-grams：

[
'for you',
'you a',
'a thousand',
'thousand times',
'times over'
]

'for you a thousand times over'

n = 3，n-grams：

[
'for you a',
'you a thousand',
'a thousand times',
'thousand times over'
]

只產生 bigram。

bigrams = CountVectorizer(ngram_range=(2,2))

產生 unigram、bigram 與 trigram。

ngrams = CountVectorizer(ngram_range=(1,3))

Python 中文本特徵工程