詞袋（Bag-of-Words）表示法
R 的自然語言處理入門
Kasey Jones
Research Data Scientist
前一個範例
animal_farm %>%
unnest_tokens(output = "word", token = "words",
input = text_column) %>%
anti_join(stop_words) %>%
count(word, sort = TRUE)
# A tibble: 3,611 x 2
word n
<chr> <int>
1 animals 248
2 farm 163
...
詞袋（Bag-of-Words）表示法
text1 <- c("Few words are important.")
text2 <- c("All words are important.")
text3 <- c("Most words are important.")
獨特詞彙：
- few：只在
text1 - all：只在
text2 - most：只在
text3 - words、are、important
常見向量表示
# Lowercase, without stop words
word_vector <- c("few", "all", "most", "words", "important")
# Representation for text1
text1 <- c("Few words are important.")
text1_vector <- c(1, 0, 0, 1, 1)
# Representation for text2
text2 <- c("All words are important.")
text2_vector <- c(0, 1, 0, 1, 1)
# Representation for text3
text3 <- c("Most words are important.")
text3_vector <- c(0, 0, 1, 1, 1)
tidytext 表示法
words <- animal_farm %>%
unnest_tokens(output = "word", token = "words", input = text_column) %>%
anti_join(stop_words) %>%
count(chapter, word, sort = TRUE)
words
# A tibble: 6,807 x 3
chapter word n
<chr> <chr> <int>
1 Chapter 8 napoleon 43
2 Chapter 8 animals 41
3 Chapter 9 boxer 34
...
單字範例
words %>%
filter(word == 'napoleon') %>%
arrange(desc(n))
# A tibble: 9 x 3
chapter word n
<chr> <chr> <int>
1 Chapter 8 napoleon 43
2 Chapter 7 napoleon 24
3 Chapter 5 napoleon 22
...
8 Chapter 3 napoleon 3
9 Chapter 4 napoleon 1
稀疏矩陣
library(tidytext); library(dplyr)
russian_tweets <- read.csv("russian_1.csv")
russian_tweets <- as_tibble(russian_tweets)
tidy_tweets <- russian_tweets %>%
unnest_tokens(word, content) %>%
anti_join(stop_words)
tidy_tweets %>%
count(word, sort = TRUE)
# A tibble: 43,666 x 2
...
稀疏矩陣（續）
稀疏矩陣
- 20,000 列（推文）
- 43,000 欄（字詞）
- 20,000 × 43,000 = 860,000,000
- 只有 177,000 個非 0 項，約 0.02%
稀疏矩陣範例：
BoW 練習
R 的自然語言處理入門
