为建模准备文本

R 自然语言处理入门

Kasey Jones

Research Data Scientist

R 中的监督学习：分类

R 自然语言处理入门

分类建模

  • 一种监督学习方法
  • 将观测分类到类别
    • 胜/负
    • 危险、友好或冷漠
  • 可用多种技术：
    • 逻辑回归
    • 决策树/随机森林/xgboost
    • 神经网络
    • 等等
R 自然语言处理入门

建模基本步骤

  1. 清洗/准备数据
  2. 划分训练集与测试集
  3. 在训练集上训练模型
  4. 在测试集上报告准确率
R 自然语言处理入门

角色识别

Napoloeon 拿破仑

拳击手 拳击手

1 https://comicvine.gamespot.com/napoleon/4005-141035/ 2 https://hero.fandom.com/wiki/Boxer_(Animal_Farm)
R 自然语言处理入门

与动物相关的句子

# Make sentences
sentences <- animal_farm %>%
  unnest_tokens(output = "sentence", token = "sentences", input = text_column)

# Label sentences by animal
sentences$boxer <- grepl('boxer', sentences$sentence)
sentences$napoleon <- grepl('napoleon', sentences$sentence)

# Replace the animal name
sentences$sentence <- gsub("boxer", "animal X", sentences$sentence)
sentences$sentence <- gsub("napoleon", "animal X", sentences$sentence)
animal_sentences <- sentences[sentences$boxer + sentences$napoleon == 1, ]
R 自然语言处理入门

句子（续）

animal_sentences$Name <-
    as.factor(ifelse(animal_sentences$boxer, "boxer", "napoleon"))

# 75 of each
animal_sentences <- 
  rbind(animal_sentences[animal_sentences$Name == "boxer", ][c(1:75), ],
        animal_sentences[animal_sentences$Name == "napoleon", ][c(1:75), ])
animal_sentences$sentence_id <- c(1:dim(animal_sentences)[1])
R 自然语言处理入门

准备数据

library(tm); library(tidytext)
library(dplyr); library(SnowballC)

animal_tokens <- animal_sentences %>%
  unnest_tokens(output = "word", token = "words", input = sentence) %>%
  anti_join(stop_words) %>%
  mutate(word = wordStem(word))
R 自然语言处理入门

准备（续）

animal_matrix <- animal_tokens %>%
  count(sentence_id, word) %>%
  cast_dtm(document = sentence_id, term = word,
           value = n, weighting = tm::weightTfIdf)
animal_matrix

<<DocumentTermMatrix (documents: 150, terms: 694)>>
Non-/sparse entries: 1235/102865
Sparsity           : 99%
Maximal term length: 17
Weighting          : term frequency - inverse document frequency
R 自然语言处理入门

移除稀疏词项

  • 非空 (1,235) + 空 (102,865)
  • 矩阵尺寸 150 × 694
  • 稀疏度：102,865 / 104,100（99%）

解决方案：removeSparseTerms()

R 自然语言处理入门

多稀疏算太稀疏？

removeSparseTerms(animal_matrix, sparse = .90)

<<DocumentTermMatrix (documents: 150, terms: 4)>>
Non-/sparse entries: 207/393
Sparsity           : 66%

removeSparseTerms(animal_matrix, sparse = .99)

removeSparseTerms(animal_matrix, sparse = .99)
<<DocumentTermMatrix (documents: 150, terms: 172)>>
Non-/sparse entries: 713/25087
Sparsity           : 97%
R 自然语言处理入门

Passons à la pratique !

R 自然语言处理入门

Preparing Video For Download...