트위터 텍스트 처리

R로 소셜 미디어 데이터 분석하기

Vivek Vijayaraghavan

Data Science Coach

학습 개요

왜 트윗 텍스트를 처리할까요?
처리 단계
- 중복/불필요 정보 제거
- 텍스트를 코퍼스로 변환
- 불용어 제거

트윗 텍스트를 처리하는 이유

트윗 텍스트는 비구조적이고 노이즈가 많습니다
이모티콘, URL, 숫자 포함
신뢰할 분석을 위해 정제가 필요합니다

텍스트 처리 단계

단계1: 불필요 정보 제거

텍스트 처리 단계

단계2: 텍스트를 코퍼스로 변환

텍스트 처리 단계

단계3: 소문자로 변환

텍스트 처리 단계

단계4: 불용어 제거

트윗 텍스트 추출

# Extract 1000 tweets on "Obesity" in English and exclude retweets
tweets_df <- search_tweets("Obesity", n = 1000, include_rts = F, lang = 'en')

# Extract the tweet texts and save it in a data frame
twt_txt <- tweets_df$text

트윗 텍스트 추출

head(twt_txt, 3)

[1] "@WeeaUwU for real, obesity should not be praised like it is in today's society" 

[2] "Great work by @DosingMatters in @AJHPOfficial on \"Vancomycin Vd estimation in 
adults with class III obesity\". As we continue to study/learn more about dosing in 
large body weight pts, we see that it's not a simple, one size, one level estimate 
that works https://t.co/KkYPqS6JzG" 

[3] "The Scottish Government have an ambition to halve childhood obesity by 2030. 
This means reducing obesity prevalence in 2-15yo children in Scotland to 7%. 
\n\n\U0001f449 In 2018, this figure was 16%\n\nFind out more in our latest blog: 
https://t.co/FWp56QWjQc https://t.co/XBK8Je7F1A"

URL 제거

# Remove URLs from the tweet text
library(qdapRegex)
twt_txt_url <- rm_twitter_url(twt_txt)

URL 제거

twt_txt_url[1:3]

[1] "@WeeaUwU for real, obesity should not be praised like it is in today's society"

[2] "Great work by @DosingMatters in @AJHPOfficial on \"Vancomycin Vd estimation in  adults 
with class III obesity\". As we continue to study/learn more about dosing in large body 
weight pts, we see that it's not a simple, one size, one level estimate that works"

[3] "The Scottish Government have an ambition to halve childhood obesity by 2030. 
This means reducing obesity prevalence in 2-15yo children in Scotland to 7%. 
\U0001f449In 2018, this figure was 16% Find out more in our latest blog:"

특수문자·구두점·숫자 제거

# Remove special characters, punctuation & numbers
twt_txt_chrs  <- gsub("[^A-Za-z]", " ", twt_txt_url)

특수문자·구두점·숫자 제거

twt_txt_chrs[1:3]

[1] " WeeaUwU for real  obesity should not be praised like it is in today s society"

[2] "Great work by  DosingMatters in  AJHPOfficial on  Vancomycin Vd estimation in 
adults with class III obesity   As we continue to study learn more about dosing in 
large body weight pts  we see that it s not a simple  one size  one level estimate 
that works"

[3] "The Scottish Government have an ambition to halve childhood obesity by       This 
means reducing obesity prevalence in     yo children in Scotland to       In       this 
figure was     Find out more in our latest blog "

텍스트를 코퍼스로 변환

# Convert to text corpus
library(tm)
twt_corpus <- twt_txt_chrs %>% 
                VectorSource() %>% 
                Corpus()

twt_corpus[[3]]$content

[1] "The Scottish Government have an ambition to halve childhood obesity by       
This means reducing obesity prevalence in     yo children in Scotland to       In  
     this figure was     Find out more in our latest blog "

소문자로 변환

대소문자만 다르면 같은 단어로 처리해야 합니다

# Convert text corpus to lowercase
twt_corpus_lwr <- tm_map(twt_corpus, tolower) 
twt_corpus_lwr[[3]]$content

[1] "the scottish government have an ambition to halve childhood obesity by     this 
means reducing obesity prevalence in     yo children in scotland to       in    this 
figure was     find out more in our latest blog "

불용어란?

불용어는 a, an, and, but 같은 자주 쓰이는 단어입니다

# Common stop words in English
stopwords("english")

영어 기본 불용어

불용어 제거

중요한 단어에 집중하려면 불용어를 제거합니다

# Remove stop words from corpus
twt_corpus_stpwd <- tm_map(twt_corpus_lwr, removeWords, stopwords("english"))

twt_corpus_stpwd[[3]]$content

[1] " scottish government   ambition  halve childhood obesity         means 
reducing obesity prevalence      yo children  scotland                figure      
find     latest blog "

추가 공백 제거

추가 공백을 제거해 코퍼스를 정리합니다

# Remove additional spaces
twt_corpus_final <- tm_map(twt_corpus_stpwd, stripWhitespace)

twt_corpus_final[[3]]$content

[1] " scottish government ambition halve childhood obesity means reducing obesity 
prevalence yo children scotland figure find latest blog "

연습해 봅시다!

R로 소셜 미디어 데이터 분석하기