실전 스크레이핑

Python으로 하는 웹 스크레이핑

Thomas Laetsch

Data Scientist, NYU

DataCamp 사이트

https://www.datacamp.com/courses/all

div가 뭐가 있나요?

# response loaded with HTML from https://www.datacamp.com/courses/all

course_divs = response.css('div.course-block')

print( len(course_divs) )
>>> 185

course-block 검사

first_div = course_divs[0]

children = first_div.xpath('./*')

print( len(children) )
>>> 3

첫 번째 자식 요소

first_div = course_divs[0]

children = first_div.xpath('./*')

first_child = children[0]

print( first_child.extract() )
>>> <a class=... />

두 번째 자식 요소

first_div = course_divs[0]

children = first_div.xpath('./*')

second_child = children[1]

print( second_child.extract() )
>>> <div class=... />

잊힌 자식 요소

first_div = course_divs[0]

children = first_div.xpath('./*')

third_child = children[2]

print( third_child.extract() )
>>> <span class=... />

리스트 만들기

하나의 CSS 선택자에서

links = response.css('div.course-block > a::attr(href)').extract()

단계별로

# step 1: course blocks
course_divs = response.css('div.course-block')

# step 2: hyperlink elements
hrefs = course_divs.xpath('./a/@href')

# step 3: extract the links
links = hrefs.extract()

코스 링크 얻기

for l in links:
    print( l )

>>> /courses/free-introduction-to-r
>>> /courses/data-table-data-manipulation-r-tutorial
>>> /courses/dplyr-data-manipulation-r-tutorial
>>> /courses/ggvis-data-visualization-r-tutorial
>>> /courses/reporting-with-r-markdown
>>> /courses/intermediate-r
...

링크 확보 완료

Python으로 하는 웹 스크레이핑