TFIDF
R 的自然語言處理入門
Kasey Jones
Research Data Scientist
Bag-of-word 的陷阱
t1 <- "My name is John. My best friend is Joe. We like tacos."
t2 <- "Two common best friend names are John and Joe."
t3 <- "Tacos are my favorite food. I eat them with my buddy Joe."
clean_t1 <- "john friend joe tacos"
clean_t2 <- "common friend john joe names"
clean_t3 <- "tacos favorite food eat buddy joe"
常見詞造成的混淆
clean_t1 <- "john friend joe tacos"
clean_t2 <- "common friend john joe names"
clean_t3 <- "tacos favorite food eat buddy joe"
比較 t1 與 t2
- t1 的 3/4 詞出現在 t2
- t2 的 3/5 詞出現在 t1
比較 t1 與 t3
- t1 的 2/4 詞出現在 t3
- t3 的 2/6 詞出現在 t1
Tacos 才關鍵
t1 <- "My name is John. My best friend is Joe. We like tacos."
t2 <- "Two common best friend names are John and Joe."
t3 <- "Tacos are my favorite food. I eat them with my friend Joe."
各文本中的詞：
- John：t1、t2
- Joe：t1、t2、t3
- Tacos：t1、t3
TFIDF
clean_t1 <- "john friend joe tacos"
clean_t2 <- "common friend john joe names"
clean_t3 <- "tacos favorite food eat buddy joe"
- TF：Term Frequency（詞頻）
- 該詞在文本中所占的比例
- john 在
clean_t1中是 1/4，tf = .25
- IDF：Inverse Document Frequency（逆文件頻率）
- 詞在整體文件中出現的普遍程度所給的權重
- john 出現在 3/3 份文件中，IDF = 0
IDF 公式
$ IDF = log \frac{N}{n_{t}} $
- N：語料庫中的文件總數
- $n_{t}$：包含該詞的文件數
例子：
- Taco 的 IDF：$log (\frac{3}{2}) = .405$
- Buddy 的 IDF：$log (\frac{3}{1}) = 1.10$
- John 的 IDF：$log (\frac{3}{3}) = 0$
TF + IDF
clean_t1 <- "john friend joe tacos"
clean_t2 <- "common friend john joe names"
clean_t3 <- "tacos favorite food eat buddy joe"
「tacos」的 TFIDF：
- clean_t1：TF * IDF = (1/4) * (.405) = 0.101
- clean_t2：TF * IDF = (0/4) * (.405) = 0
- clean_t3：TF * IDF = (1/6) * (.405) = 0.068
計算 TFIDF 矩陣
# Create a data.frame
df <- data.frame('text' = c(t1, t2, t3), 'ID' = c(1, 2, 3))
df %>%
unnest_tokens(output = "word", token = "words", input = text) %>%
anti_join(stop_words) %>%
count(ID, word, sort = TRUE) %>%
bind_tf_idf(word, ID, n)
- word：包含詞項的欄位
- ID：包含文件 ID 的欄位
- n：
count()產生的詞頻
bind_tf_idf 輸出
# A tibble: 15 x 6
X word n tf idf tf_idf
<dbl> <chr> <int> <dbl> <dbl> <dbl>
1 1 friend 1 0.25 0.405 0.101
2 1 joe 1 0.25 0 0
3 1 john 1 0.25 0.405 0.101
4 1 tacos 1 0.25 0.405 0.101
5 2 common 1 0.2 1.10 0.220
6 2 friend 1 0.2 0.405 0.0811
...
TFIDF 練習
R 的自然語言處理入門
