서비스 요청

Python으로 하는 웹 스크레이핑

Thomas Laetsch

Data Scientist, NYU

스파이더 복습

import scrapy
from scrapy.crawler import CrawlerProcess

class SpiderClassName(scrapy.Spider):
    name = "spider_name"
    # the code for your spider
    ...

process = CrawlerProcess()

process.crawl(SpiderClassName)

process.start()

스파이더 복습

class DCspider( scrapy.Spider ):
    name = "dc_spider"

    def start_requests( self ):
        urls = [ 'https://www.datacamp.com/courses/all' ]
        for url in urls:
            yield scrapy.Request( url = url, callback = self.parse )

    def parse( self, response ):
        # simple example: write out the html
        html_file = 'DC_courses.html'
        with open( html_file, 'wb' ) as fout:
            fout.write( response.body )

start_requests 핵심

def start_requests( self ):

    urls = ['https://www.datacamp.com/courses/all']

    for url in urls:
        yield scrapy.Request( url = url, callback = self.parse )

def start_requests( self ):
    url = 'https://www.datacamp.com/courses/all'
    yield scrapy.Request( url = url, callback = self.parse )

여기서 scrapy.Request는 응답 변수를 채워 줍니다
url 인수는 크롤링할 사이트를 지정합니다
callback 인수는 응답 변수를 보낼 처리 함수를 지정합니다

확대 보기

class DCspider( scrapy.Spider ):
    name = "dc_spider"

    def start_requests( self ):
        urls = [ 'https://www.datacamp.com/courses/all' ]
        for url in urls:
            yield scrapy.Request( url = url, callback = self.parse )

    def parse( self, response ):
        # simple example: write out the html
        html_file = 'DC_courses.html'
        with open( html_file, 'wb' ) as fout:
            fout.write( response.body )

요청 종료

Python으로 하는 웹 스크레이핑