Tokenização

Introdução ao Processamento de Linguagem Natural em R

Kasey Jones

Research Data Scientist

O que são tokens?

Tipos comuns de tokenização:

caracteres
palavras
sentenças
documentos
separações por expressões regulares

Pacote tidytext

Visão geral do pacote:

"Text Mining using dplyr, ggplot2, and Other Tidy Tools"
Segue o formato de dados tidy

Introduction to the Tidyverse

¹ https://cran.r-project.org/web/packages/tidytext/index.html

O conjunto de dados de Animal Farm

animal_farm

# A tibble: 10 x 2
   chapter    text_column                                                                                                      
   <chr>      <chr>                                                                                                            
 1 Chapter 1  "Mr. Jones, of the Manor Farm, had locked ...
 2 Chapter 2  "Three nights later old Major died peacefully ...
 3 Chapter 3  "How they toiled and sweated to get the hay ...
...

¹ https://en.wikipedia.org/wiki/Animal_Farm

Praticando tokenização

animal_farm %>%
  unnest_tokens(output = "word",
                input = text_column,
                token = "words")

Opções de token

sentences
lines
regex
words
...

Contando tokens

animal_farm %>%
  unnest_tokens(output = "word",
                token = "words",
                input = text_column) %>%
  count(word, sort = TRUE)

# A tibble: 4,076 x 2
   word      n
   <chr> <int>
 1 the    2187
 2 and     966
 3 of      899
 4 to      814
 ...

Tokenização com expressões regulares

animal_farm %>%
  filter(chapter == 'Chapter 1') %>%
  unnest_tokens(output = "Boxer", input = text_column,
                token = "regex", pattern = "(?i)boxer") %>%
  slice(2:n())

# A tibble: 5 x 2
  chapter   Boxer                                                                                                                                 
  <chr>     <chr>                                                                                                                                 
2 Chapter 1 " and clover, came in together, walking very slowly and setting down their vast hairy hoofs with great care lest there should be some…
3 Chapter 1 " was an enormous beast, nearly eighteen hands high, and as strong as any two ordinary horses put together. a white stripe down his n…
4 Chapter 1 "; the two of them usually spent their sundays together in the small paddock beyond the orchard, grazing side by side and never speak…
...

Vamos tokenizar um texto.

Introdução ao Processamento de Linguagem Natural em R