词袋表示法
R 自然语言处理入门
Kasey Jones
Research Data Scientist
前面的示例
animal_farm %>%
unnest_tokens(output = "word", token = "words",
input = text_column) %>%
anti_join(stop_words) %>%
count(word, sort = TRUE)
# A tibble: 3,611 x 2
word n
<chr> <int>
1 animals 248
2 farm 163
...
词袋表示法
text1 <- c("Few words are important.")
text2 <- c("All words are important.")
text3 <- c("Most words are important.")
唯一词：
- few：仅在
text1中 - all：仅在
text2中 - most：仅在
text3中 - words、are、important
常见向量表示
# 小写，去停用词
word_vector <- c("few", "all", "most", "words", "important")
# text1 的表示
text1 <- c("Few words are important.")
text1_vector <- c(1, 0, 0, 1, 1)
# text2 的表示
text2 <- c("All words are important.")
text2_vector <- c(0, 1, 0, 1, 1)
# text3 的表示
text3 <- c("Most words are important.")
text3_vector <- c(0, 0, 1, 1, 1)
tidytext 表示
words <- animal_farm %>%
unnest_tokens(output = "word", token = "words", input = text_column) %>%
anti_join(stop_words) %>%
count(chapter, word, sort = TRUE)
words
# A tibble: 6,807 x 3
chapter word n
<chr> <chr> <int>
1 Chapter 8 napoleon 43
2 Chapter 8 animals 41
3 Chapter 9 boxer 34
...
单词示例
words %>%
filter(word == 'napoleon') %>%
arrange(desc(n))
# A tibble: 9 x 3
chapter word n
<chr> <chr> <int>
1 Chapter 8 napoleon 43
2 Chapter 7 napoleon 24
3 Chapter 5 napoleon 22
...
8 Chapter 3 napoleon 3
9 Chapter 4 napoleon 1
稀疏矩阵
library(tidytext); library(dplyr)
russian_tweets <- read.csv("russian_1.csv")
russian_tweets <- as_tibble(russian_tweets)
tidy_tweets <- russian_tweets %>%
unnest_tokens(word, content) %>%
anti_join(stop_words)
tidy_tweets %>%
count(word, sort = TRUE)
# A tibble: 43,666 x 2
...
稀疏矩阵（续）
稀疏矩阵
- 20,000 行（推文）
- 43,000 列（词）
- 20,000 × 43,000 = 860,000,000
- 仅 177,000 个非 0 项，约 0.02%
稀疏矩阵示例：
BoW 练习
R 自然语言处理入门
