Stop Scratching and Start Scraping!

Web Scraping in Python

Thomas Laetsch

Data Scientist, NYU

Feeding the Machine

Scraping Skills

• Objective: Scrape a website computationally
• How? We decide to use scrapy
• How? We need to work with:
  • Selector and Response objects
  • Maybe even create a Spider
• How? We need to learn XPath or CSS Locator notation
• How? Understand the structure of HTML

What'd'ya Know?

• Structure of HTML
• XPath and CSS Locator notation
• How to use Selector and Response objects in scrapy
• How to set up a spider
• How to scrape the web

EOT

Web Scraping in Python