Attribute and Text Selection

Web Scraping in Python

Thomas Laetsch

Data Scientist, NYU

You Must have Guts to use your Colon

  • Using XPath: <xpath-to-element>/@attr-name
xpath = '//div[@id="uid"]/a/@href'
  • Using CSS Locator: <css-to-element>::attr(attr-name)
css_locator = 'div#uid > a::attr(href)'
Web Scraping in Python

Text Extraction

<p id="p-example">
  Hello world! 
  Try <a href="http://www.datacamp.com">DataCamp</a> today!
</p>
  • In XPath use text()
sel.xpath('//p[@id="p-example"]/text()').extract()

# result: ['\n Hello world!\n Try ', ' today!\n']

sel.xpath('//p[@id="p-example"]//text()').extract()

# result: ['\n Hello world!\n Try ', 'DataCamp', ' today!\n']
Web Scraping in Python

Text Extraction

<p id="p-example">
  Hello world! 
  Try <a href="http://www.datacamp.com">DataCamp</a> today!
</p>
  • For CSS Locator, use ::text
sel.css('p#p-example::text').extract()

# result: ['\n Hello world!\n Try ', ' today!\n']

sel.css('p#p-example ::text').extract()

# result: ['\n Hello world!\n Try ', 'DataCamp', ' today!\n']
Web Scraping in Python

Scoping the Colon

Web Scraping in Python

Preparing Video For Download...