Tokenleştirme

R ile Doğal Dil İşlemeye Giriş

Kasey Jones

Research Data Scientist

Belirteçler nedir?

Yaygın tokenleştirme türleri:

karakterler
kelimeler
cümleler
belgeler
düzenli ifade ayırmaları

tidytext paketi

Paket özeti:

"dplyr, ggplot2 ve diğer Tidy araçlarıyla Metin Madenciliği"
Tidy veri biçimini izler

Introduction to the Tidyverse

¹ https://cran.r-project.org/web/packages/tidytext/index.html

Animal Farm veri seti

animal_farm

# A tibble: 10 x 2
   chapter    text_column                                                                                                      
   <chr>      <chr>                                                                                                            
 1 Chapter 1  "Mr. Jones, of the Manor Farm, had locked ...
 2 Chapter 2  "Three nights later old Major died peacefully ...
 3 Chapter 3  "How they toiled and sweated to get the hay ...
...

¹ https://en.wikipedia.org/wiki/Animal_Farm

Tokenleştirme alıştırması

animal_farm %>%
  unnest_tokens(output = "word",
                input = text_column,
                token = "words")

Belirteç seçenekleri

sentences
lines
regex
words
...

Belirteçleri sayma

animal_farm %>%
  unnest_tokens(output = "word",
                token = "words",
                input = text_column) %>%
  count(word, sort = TRUE)

# A tibble: 4,076 x 2
   word      n
   <chr> <int>
 1 the    2187
 2 and     966
 3 of      899
 4 to      814
 ...

Düzenli ifadelerle tokenleştirme

animal_farm %>%
  filter(chapter == 'Chapter 1') %>%
  unnest_tokens(output = "Boxer", input = text_column,
                token = "regex", pattern = "(?i)boxer") %>%
  slice(2:n())

# A tibble: 5 x 2
  chapter   Boxer                                                                                                                                 
  <chr>     <chr>                                                                                                                                 
2 Chapter 1 " and clover, came in together, walking very slowly and setting down their vast hairy hoofs with great care lest there should be some…
3 Chapter 1 " was an enormous beast, nearly eighteen hands high, and as strong as any two ordinary horses put together. a white stripe down his n…
4 Chapter 1 "; the two of them usually spent their sundays together in the small paddock beyond the orchard, grazing side by side and never speak…
...

Biraz metin tokenleştirelim.

R ile Doğal Dil İşlemeye Giriş