n-gram 모델 구축

Python으로 배우는 NLP 피처 엔지니어링

Rounak Banik

Data Scientist

BoW의 한계

review	label
`'The movie was good and not boring'`	positive
`'The movie was not good and boring'`	negative

n = 2, n-gram:

[
'for you',
'you a',
'a thousand',
'thousand times',
'times over'
]

'for you a thousand times over'

n = 3, n-gram:

[
'for you a',
'you a thousand',
'a thousand times',
'thousand times over'
]

바이그램만 생성.

bigrams = CountVectorizer(ngram_range=(2,2))

유니그램, 바이그램, 트라이그램 생성.

ngrams = CountVectorizer(ngram_range=(1,3))

Python으로 배우는 NLP 피처 엔지니어링