处理 Twitter 文本

用 R 分析社交媒体数据

Vivek Vijayaraghavan

Data Science Coach

课程概览

  • 为什么处理推文文本？
  • 处理步骤
    • 去除冗余信息
    • 将文本转换为语料库
    • 移除停用词
用 R 分析社交媒体数据

为何处理推文文本？

  • 推文文本非结构化、噪声多、原始
  • 包含表情、URL、数字
  • 为了分析与可靠结果，需要清洗文本
用 R 分析社交媒体数据

文本处理步骤

步骤1：去除冗余信息

用 R 分析社交媒体数据

文本处理步骤

步骤2：将文本转换为语料库

用 R 分析社交媒体数据

文本处理步骤

步骤3：转换为小写

用 R 分析社交媒体数据

文本处理步骤

步骤4：移除停用词

用 R 分析社交媒体数据

提取推文文本

# Extract 1000 tweets on "Obesity" in English and exclude retweets
tweets_df <- search_tweets("Obesity", n = 1000, include_rts = F, lang = 'en')

# Extract the tweet texts and save it in a data frame
twt_txt <- tweets_df$text
用 R 分析社交媒体数据

提取推文文本

head(twt_txt, 3)

[1] "@WeeaUwU for real, obesity should not be praised like it is in today's society" 

[2] "Great work by @DosingMatters in @AJHPOfficial on \"Vancomycin Vd estimation in 
adults with class III obesity\". As we continue to study/learn more about dosing in 
large body weight pts, we see that it's not a simple, one size, one level estimate 
that works https://t.co/KkYPqS6JzG" 

[3] "The Scottish Government have an ambition to halve childhood obesity by 2030. 
This means reducing obesity prevalence in 2-15yo children in Scotland to 7%. 
\n\n\U0001f449 In 2018, this figure was 16%\n\nFind out more in our latest blog: 
https://t.co/FWp56QWjQc https://t.co/XBK8Je7F1A"
用 R 分析社交媒体数据

移除 URL

# Remove URLs from the tweet text
library(qdapRegex)
twt_txt_url <- rm_twitter_url(twt_txt)
用 R 分析社交媒体数据

移除 URL

twt_txt_url[1:3]

[1] "@WeeaUwU for real, obesity should not be praised like it is in today's society"

[2] "Great work by @DosingMatters in @AJHPOfficial on \"Vancomycin Vd estimation in  adults 
with class III obesity\". As we continue to study/learn more about dosing in large body 
weight pts, we see that it's not a simple, one size, one level estimate that works"

[3] "The Scottish Government have an ambition to halve childhood obesity by 2030. 
This means reducing obesity prevalence in 2-15yo children in Scotland to 7%. 
\U0001f449In 2018, this figure was 16% Find out more in our latest blog:"
用 R 分析社交媒体数据

特殊字符、标点与数字

# Remove special characters, punctuation & numbers
twt_txt_chrs  <- gsub("[^A-Za-z]", " ", twt_txt_url)
用 R 分析社交媒体数据

特殊字符、标点与数字

twt_txt_chrs[1:3]

[1] " WeeaUwU for real  obesity should not be praised like it is in today s society"

[2] "Great work by  DosingMatters in  AJHPOfficial on  Vancomycin Vd estimation in 
adults with class III obesity   As we continue to study learn more about dosing in 
large body weight pts  we see that it s not a simple  one size  one level estimate 
that works"

[3] "The Scottish Government have an ambition to halve childhood obesity by       This 
means reducing obesity prevalence in     yo children in Scotland to       In       this 
figure was     Find out more in our latest blog "
用 R 分析社交媒体数据

转换为文本语料库

# Convert to text corpus
library(tm)
twt_corpus <- twt_txt_chrs %>% 
                VectorSource() %>% 
                Corpus()

twt_corpus[[3]]$content

[1] "The Scottish Government have an ambition to halve childhood obesity by       
This means reducing obesity prevalence in     yo children in Scotland to       In  
     this figure was     Find out more in our latest blog "
用 R 分析社交媒体数据

转换为小写

  • 同一词不应因大小写不同而被计为两个词
# Convert text corpus to lowercase
twt_corpus_lwr <- tm_map(twt_corpus, tolower) 
twt_corpus_lwr[[3]]$content

[1] "the scottish government have an ambition to halve childhood obesity by     this 
means reducing obesity prevalence in     yo children in scotland to       in    this 
figure was     find out more in our latest blog "
用 R 分析社交媒体数据

什么是停用词？

  • 停用词是常见词，如 a、an、and、but
# Common stop words in English
stopwords("english")

英语默认停用词

用 R 分析社交媒体数据

移除停用词

  • 为聚焦关键信息，需要移除停用词
# Remove stop words from corpus
twt_corpus_stpwd <- tm_map(twt_corpus_lwr, removeWords, stopwords("english"))

twt_corpus_stpwd[[3]]$content


[1] " scottish government   ambition  halve childhood obesity         means 
reducing obesity prevalence      yo children  scotland                figure      
find     latest blog "
用 R 分析社交媒体数据

移除多余空格

  • 去除多余空格以得到干净语料
# Remove additional spaces
twt_corpus_final <- tm_map(twt_corpus_stpwd, stripWhitespace)

twt_corpus_final[[3]]$content

[1] " scottish government ambition halve childhood obesity means reducing obesity 
prevalence yo children scotland figure find latest blog "
用 R 分析社交媒体数据

Passons à la pratique !

用 R 分析社交媒体数据

Preparing Video For Download...