用 Python 擷取網頁

Python 資料匯入進階

Hugo Bowne-Anderson

Data Scientist at DataCamp

HTML

  • 混合非結構化與結構化資料

  • 結構化資料：

    • 具預先定義的資料模型，或

    • 以固定方式組織

  • 非結構化資料：不具上述性質

ch_1_3.008.png

Python 資料匯入進階

BeautifulSoup

  • 從 HTML 解析並擷取結構化資料

ch_1_3.011.png

  • 將雜亂的標籤整理美化並擷取資訊
Python 資料匯入進階

BeautifulSoup

from bs4 import BeautifulSoup
import requests
url = 'https://www.crummy.com/software/BeautifulSoup/'
r = requests.get(url)
html_doc = r.text
soup = BeautifulSoup(html_doc)
Python 資料匯入進階

美化後的 Soup

print(soup.prettify())

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/transitional.dtd">
<html>
 <head>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
  <title>
   Beautiful Soup: We called him Tortoise because he taught us.
  </title>
  <link href="mailto:[email protected]" rev="made"/>
  <link href="/nb/themes/Default/nb.css" rel="stylesheet" type="text/css"/>
  <meta content="Beautiful Soup: a library designed for screen-scraping HTML and XML." name="Description"/>
  <meta content="Markov Approximation 1.4 (module: leonardr)" name="generator"/>
  <meta content="Leonard Richardson" name="author"/>
 </head>
 <body alink="red" bgcolor="white" link="blue" text="black" vlink="660066">
  <img align="right" src="10.1.jpg" width="250"/>
  <br/>
  <p>
Python 資料匯入進階

探索 BeautifulSoup

  • 多種方法，例如：
print(soup.title)

<title>Beautiful Soup: We called him Tortoise because he taught us.</title>

print(soup.get_text())

Beautiful Soup: We called him Tortoise because he taught us.
You didn't write that awful page. You're just trying to
get some data out of it. Beautiful Soup is here to 
help. Since 2004, it's been saving programmers hours or
days of work on quick-turnaround screen scraping 
projects.
Python 資料匯入進階

探索 BeautifulSoup

  • find_all()
for link in soup.find_all('a'):
    print(link.get('href'))

bs4/download/
#Download
bs4/doc/
#HallOfFame
https://code.launchpad.net/beautifulsoup
https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup
http://www.candlemarkandgleam.com/shop/constellation-games/
http://constellation.crummy.com/Constellation%20Games%20excerpt.html
https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup
https://bugs.launchpad.net/beautifulsoup/
http://lxml.de/
http://code.google.com/p/html5lib/
Python 資料匯入進階

一起來練習吧！

Python 資料匯入進階

Preparing Video For Download...