トークン化

Rで学ぶ自然言語処理入門

Kasey Jones

Research Data Scientist

トークンとは？

一般的なトークン化の種類:

文字
単語
文
文書
正規表現での分割

tidytext パッケージ

パッケージ概要:

「dplyr、ggplot2、その他のtidyツールでのテキストマイニング」
tidyデータ形式に従う

Introduction to the Tidyverse

¹ https://cran.r-project.org/web/packages/tidytext/index.html

Animal Farm データセット

animal_farm

# A tibble: 10 x 2
   chapter    text_column                                                                                                      
   <chr>      <chr>                                                                                                            
 1 Chapter 1  "Mr. Jones, of the Manor Farm, had locked ...
 2 Chapter 2  "Three nights later old Major died peacefully ...
 3 Chapter 3  "How they toiled and sweated to get the hay ...
...

¹ https://en.wikipedia.org/wiki/Animal_Farm

トークン化の練習

animal_farm %>%
  unnest_tokens(output = "word",
                input = text_column,
                token = "words")

トークンのオプション

sentences
lines
regex
words
...

トークンのカウント

animal_farm %>%
  unnest_tokens(output = "word",
                token = "words",
                input = text_column) %>%
  count(word, sort = TRUE)

# A tibble: 4,076 x 2
   word      n
   <chr> <int>
 1 the    2187
 2 and     966
 3 of      899
 4 to      814
 ...

正規表現でのトークン化

animal_farm %>%
  filter(chapter == 'Chapter 1') %>%
  unnest_tokens(output = "Boxer", input = text_column,
                token = "regex", pattern = "(?i)boxer") %>%
  slice(2:n())

# A tibble: 5 x 2
  chapter   Boxer                                                                                                                                 
  <chr>     <chr>                                                                                                                                 
2 Chapter 1 " and clover, came in together, walking very slowly and setting down their vast hairy hoofs with great care lest there should be some…
3 Chapter 1 " was an enormous beast, nearly eighteen hands high, and as strong as any two ordinary horses put together. a white stripe down his n…
4 Chapter 1 "; the two of them usually spent their sundays together in the small paddock beyond the orchard, grazing side by side and never speak…
...

テキストをトークン化してみましょう。

Rで学ぶ自然言語処理入門