토큰화

R로 배우는 자연어 처리 입문

Kasey Jones

Research Data Scientist

토큰이란?

일반적인 토큰화 유형:

문자
단어
문장
문서
정규식 분리

tidytext 패키지

패키지 개요:

"dplyr, ggplot2 등 타이디 도구로 텍스트 마이닝"
타이디 데이터 형식을 따름

Introduction to the Tidyverse

¹ https://cran.r-project.org/web/packages/tidytext/index.html

Animal Farm 데이터셋

animal_farm

# A tibble: 10 x 2
   chapter    text_column                                                                                                      
   <chr>      <chr>                                                                                                            
 1 Chapter 1  "Mr. Jones, of the Manor Farm, had locked ...
 2 Chapter 2  "Three nights later old Major died peacefully ...
 3 Chapter 3  "How they toiled and sweated to get the hay ...
...

¹ https://en.wikipedia.org/wiki/Animal_Farm

토큰화 연습

animal_farm %>%
  unnest_tokens(output = "word",
                input = text_column,
                token = "words")

토큰 옵션

sentences
lines
regex
words
...

토큰 개수 세기

animal_farm %>%
  unnest_tokens(output = "word",
                token = "words",
                input = text_column) %>%
  count(word, sort = TRUE)

# A tibble: 4,076 x 2
   word      n
   <chr> <int>
 1 the    2187
 2 and     966
 3 of      899
 4 to      814
 ...

정규식으로 토큰화

animal_farm %>%
  filter(chapter == 'Chapter 1') %>%
  unnest_tokens(output = "Boxer", input = text_column,
                token = "regex", pattern = "(?i)boxer") %>%
  slice(2:n())

# A tibble: 5 x 2
  chapter   Boxer                                                                                                                                 
  <chr>     <chr>                                                                                                                                 
2 Chapter 1 " and clover, came in together, walking very slowly and setting down their vast hairy hoofs with great care lest there should be some…
3 Chapter 1 " was an enormous beast, nearly eighteen hands high, and as strong as any two ordinary horses put together. a white stripe down his n…
4 Chapter 1 "; the two of them usually spent their sundays together in the small paddock beyond the orchard, grazing side by side and never speak…
...

텍스트를 토큰화해 봅시다.

R로 배우는 자연어 처리 입문