Twitterテキストの処理

Rで学ぶソーシャルメディアデータ分析

Vivek Vijayaraghavan

Data Science Coach

レッスン概要

なぜツイート本文を処理するのか
テキスト処理の手順
- 冗長情報の除去
- コーパスへ変換
- ストップワードの除去

なぜツイート本文を処理するか

ツイート本文は非構造・ノイズ多・生データ
絵文字、URL、数字を含む
解析と信頼できる結果にはクリーン化が必要

テキスト処理の手順

手順1：冗長情報を削除

テキスト処理の手順

手順2：テキストをコーパスへ

テキスト処理の手順

手順3：小文字に変換

テキスト処理の手順

手順4：ストップワードを除去

ツイート本文を抽出

# Extract 1000 tweets on "Obesity" in English and exclude retweets
tweets_df <- search_tweets("Obesity", n = 1000, include_rts = F, lang = 'en')

# Extract the tweet texts and save it in a data frame
twt_txt <- tweets_df$text

ツイート本文を抽出

head(twt_txt, 3)

[1] "@WeeaUwU for real, obesity should not be praised like it is in today's society" 

[2] "Great work by @DosingMatters in @AJHPOfficial on \"Vancomycin Vd estimation in 
adults with class III obesity\". As we continue to study/learn more about dosing in 
large body weight pts, we see that it's not a simple, one size, one level estimate 
that works https://t.co/KkYPqS6JzG" 

[3] "The Scottish Government have an ambition to halve childhood obesity by 2030. 
This means reducing obesity prevalence in 2-15yo children in Scotland to 7%. 
\n\n\U0001f449 In 2018, this figure was 16%\n\nFind out more in our latest blog: 
https://t.co/FWp56QWjQc https://t.co/XBK8Je7F1A"

URLの除去

# Remove URLs from the tweet text
library(qdapRegex)
twt_txt_url <- rm_twitter_url(twt_txt)

URLの除去

twt_txt_url[1:3]

[1] "@WeeaUwU for real, obesity should not be praised like it is in today's society"

[2] "Great work by @DosingMatters in @AJHPOfficial on \"Vancomycin Vd estimation in  adults 
with class III obesity\". As we continue to study/learn more about dosing in large body 
weight pts, we see that it's not a simple, one size, one level estimate that works"

[3] "The Scottish Government have an ambition to halve childhood obesity by 2030. 
This means reducing obesity prevalence in 2-15yo children in Scotland to 7%. 
\U0001f449In 2018, this figure was 16% Find out more in our latest blog:"

記号・句読点・数字の除去

# Remove special characters, punctuation & numbers
twt_txt_chrs  <- gsub("[^A-Za-z]", " ", twt_txt_url)

記号・句読点・数字の除去

twt_txt_chrs[1:3]

[1] " WeeaUwU for real  obesity should not be praised like it is in today s society"

[2] "Great work by  DosingMatters in  AJHPOfficial on  Vancomycin Vd estimation in 
adults with class III obesity   As we continue to study learn more about dosing in 
large body weight pts  we see that it s not a simple  one size  one level estimate 
that works"

[3] "The Scottish Government have an ambition to halve childhood obesity by       This 
means reducing obesity prevalence in     yo children in Scotland to       In       this 
figure was     Find out more in our latest blog "

テキストをコーパスに変換

# Convert to text corpus
library(tm)
twt_corpus <- twt_txt_chrs %>% 
                VectorSource() %>% 
                Corpus()

twt_corpus[[3]]$content

[1] "The Scottish Government have an ambition to halve childhood obesity by       
This means reducing obesity prevalence in     yo children in Scotland to       In  
     this figure was     Find out more in our latest blog "

小文字に変換

大文字小文字が違っても同じ語は1語として数えるべき

# Convert text corpus to lowercase
twt_corpus_lwr <- tm_map(twt_corpus, tolower) 
twt_corpus_lwr[[3]]$content

[1] "the scottish government have an ambition to halve childhood obesity by     this 
means reducing obesity prevalence in     yo children in scotland to       in    this 
figure was     find out more in our latest blog "

ストップワードとは

ストップワードは a, an, and, but など頻出の機能語

# Common stop words in English
stopwords("english")

英語のデフォルトのストップワード

ストップワードを除去

重要語に集中するためストップワードを除去する

# Remove stop words from corpus
twt_corpus_stpwd <- tm_map(twt_corpus_lwr, removeWords, stopwords("english"))

twt_corpus_stpwd[[3]]$content

[1] " scottish government   ambition  halve childhood obesity         means 
reducing obesity prevalence      yo children  scotland                figure      
find     latest blog "

余分な空白を削除

余分な空白を削除し、クリーンなコーパスにする

# Remove additional spaces
twt_corpus_final <- tm_map(twt_corpus_stpwd, stripWhitespace)

twt_corpus_final[[3]]$content

[1] " scottish government ambition halve childhood obesity means reducing obesity 
prevalence yo children scotland figure find latest blog "

練習に進みましょう！

Rで学ぶソーシャルメディアデータ分析