要求を穏やかにし、リクエストを減速する方法

Rで学ぶWebスクレイピング

Timo Grossenbacher

Instructor

絶対に真似しないでください！

library(httr)
while(TRUE){
  print(Sys.time())
  response <- 
    GET("https://httpbin.org")
  print(status_code(response))
}

[1] "2020-06-20 10:31:17 CEST"
[1] 200
[1] "2020-06-20 10:31:17 CEST"
[1] 200
[1] "2020-06-20 10:31:17 CEST"
[1] 200
[1] "2020-06-20 10:31:17 CEST"
[1] 200
[1] "2020-06-20 10:31:17 CEST"
[1] 200
[1] "2020-06-20 10:31:18 CEST"
[1] 200
...

より丁寧なWebサイトへのリクエスト方法

while(TRUE){
  # 1秒待機
  # ...
  print(Sys.time())
  response <- 
    GET("https://httpbin.org")
  print(status_code(response))
}

[1] "2020-06-20 10:36:06 CEST"
[1] 200
[1] "2020-06-20 10:36:07 CEST"
[1] 200
[1] "2020-06-20 10:36:08 CEST"
[1] 200
[1] "2020-06-20 10:36:09 CEST"
[1] 200
[1] "2020-06-20 10:36:10 CEST"
[1] 200
[1] "2020-06-20 10:36:11 CEST"
[1] 200
...

tidyなスロットリング手法

関数のスロットリング＝呼び出し間に待機時間を入れる

library(httr)
library(purrr)
throttled_GET <- slowly(
  ~ GET("https://httbin.org"),

    rate = rate_delay(3))

while(TRUE){
  print(Sys.time())
  response <- throttled_GET()
  print(status_code(response))
}

[1] "2020-06-20 10:53:44 CEST"
[1] 200
[1] "2020-06-20 10:53:47 CEST"
[1] 200
[1] "2020-06-20 10:53:50 CEST"
[1] 200
[1] "2020-06-20 10:53:53 CEST"
[1] 200
[1] "2020-06-20 10:53:56 CEST"
[1] 200
...

スロットル関数でカスタムURLを照会する

library(httr)
library(purrr)
throttled_GET <-
    # GET("https://...") の代わりに
    slowly(~ GET(.), rate = rate_delay(3))

while(TRUE){
  print(Sys.time())
  response <- 
      throttled_GET("https://wikipedia.org")
  print(status_code(response))
}

[1] "2020-06-20 10:53:44 CEST"
[1] 200
[1] "2020-06-20 10:53:47 CEST"
[1] 200
[1] "2020-06-20 10:53:50 CEST"
[1] 200
[1] "2020-06-20 10:53:53 CEST"
[1] 200
[1] "2020-06-20 10:53:56 CEST"
[1] 200
...

URLリストをループ処理する

library(httr)
url_list <- c("https://httbin.org/anything/1",
              "https://httbin.org/anything/2",
              "https://httbin.org/anything/3")

for(url in url_list){
  response <- throttled_GET(url)
  print(status_code(response))
}

[1] 200
[1] 200
[1] 200

library(httr)
url_list <- c("https://wikipedia.org/wiki/K2",
              "https://wikipedia.org/wiki/\
    Mount_Everest")

for(url in url_list){
  response <- throttled_GET(url)
  print(status_code(response))
}

[1] 200
[1] 200

実例に適用してみましょう！

Rで学ぶWebスクレイピング