Telling who you are with custom user agents

Web Scraping in R

Timo Grossenbacher

Instructor

Show yourself!

  • Web server already registers your IP address
  • Better to explicitly identify yourself
  • There's an HTTP header for that!

An example request

1 https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview
Web Scraping in R

Modify headers with httr

response <- GET('http://example.com', user_agent("Hey, it's me, Timo! 
    Reach me at [email protected]."))

Alternatively:

set_config(add_headers(`User-Agent` = "Hey, it's me, Timo!
    Reach me at [email protected]."))
response <- GET('http://example.com')
Web Scraping in R

Let's try this!

Web Scraping in R

Preparing Video For Download...