파이썬으로 웹 스크래핑

Intermediate Importing Data in Python

Hugo Bowne-Anderson

Data Scientist at DataCamp

HTML

  • 비정형 + 정형 데이터 혼합

  • 정형 데이터:

    • 미리 정의된 데이터 모델이 있음, 또는

    • 정해진 방식으로 구성됨

  • 비정형 데이터: 위 특성이 없음

ch_1_3.008.png

Intermediate Importing Data in Python

BeautifulSoup

  • HTML에서 정형 데이터를 파싱·추출

ch_1_3.011.png

  • 난잡한 태그를 정리하고 정보 추출
Intermediate Importing Data in Python

BeautifulSoup

from bs4 import BeautifulSoup
import requests
url = 'https://www.crummy.com/software/BeautifulSoup/'
r = requests.get(url)
html_doc = r.text
soup = BeautifulSoup(html_doc)
Intermediate Importing Data in Python

정돈된 Soup 출력

print(soup.prettify())

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/transitional.dtd">
<html>
 <head>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
  <title>
   Beautiful Soup: We called him Tortoise because he taught us.
  </title>
  <link href="mailto:[email protected]" rev="made"/>
  <link href="/nb/themes/Default/nb.css" rel="stylesheet" type="text/css"/>
  <meta content="Beautiful Soup: a library designed for screen-scraping HTML and XML." name="Description"/>
  <meta content="Markov Approximation 1.4 (module: leonardr)" name="generator"/>
  <meta content="Leonard Richardson" name="author"/>
 </head>
 <body alink="red" bgcolor="white" link="blue" text="black" vlink="660066">
  <img align="right" src="10.1.jpg" width="250"/>
  <br/>
  <p>
Intermediate Importing Data in Python

BeautifulSoup 살펴보기

  • 다양한 메서드 예:
print(soup.title)

<title>Beautiful Soup: We called him Tortoise because he taught us.</title>

print(soup.get_text())

Beautiful Soup: We called him Tortoise because he taught us.
You didn't write that awful page. You're just trying to
get some data out of it. Beautiful Soup is here to 
help. Since 2004, it's been saving programmers hours or
days of work on quick-turnaround screen scraping 
projects.
Intermediate Importing Data in Python

BeautifulSoup 살펴보기

  • find_all()
for link in soup.find_all('a'):
    print(link.get('href'))

bs4/download/
#Download
bs4/doc/
#HallOfFame
https://code.launchpad.net/beautifulsoup
https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup
http://www.candlemarkandgleam.com/shop/constellation-games/
http://constellation.crummy.com/Constellation%20Games%20excerpt.html
https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup
https://bugs.launchpad.net/beautifulsoup/
http://lxml.de/
http://code.google.com/p/html5lib/
Intermediate Importing Data in Python

연습해 봅시다!

Intermediate Importing Data in Python

Preparing Video For Download...