并行调试

R 并行编程

Nabeel Imam

Data Scientist

什么是调试？

程序员在电脑屏幕上发现并尝试移除一个代码缺陷。

并行读取文件

print(file_list)

 [1] "./stocks/2011.csv"
 [2] "./stocks/2012.csv"
 [3] "./stocks/2013.csv"
 [4] "./stocks/2014.csv"
 [5] "./stocks/2015.csv"
 ...

过滤函数

filterCSV <- function (filepath) {

  # Read CSV
  df <- read.csv(filepath)

  # Filter data
  df <- df %>%
    dplyr::filter(Company == "Tesla")

  # Write to back to same path
  write.csv(df, filepath)
}

并行 apply

cl <- makeCluster(4)

clusterEvalQ(cl, library(dplyr))


dummy <- parLapply(cl, file_list, filterCSV)

stopCluster(cl)

Error in checkForRemoteErrors(val) : 
  one node produced an error: ℹ In argument: `Company == "Tesla"`.
Caused by error:
! object 'Company' not found

顺序运行

short_list <- file_list[1:5]


dummy <- lapply(short_list, filterCSV)


read.csv(short_list[1])

         Date  Open  High   Low Close Adj.Close   Volume Company Year
1  2011-01-03 5.368 5.400 5.180 5.324     5.324  6415000   Tesla 2011
2  2011-01-04 5.332 5.390 5.204 5.334     5.334  5937000   Tesla 2011
3  2011-01-05 5.296 5.380 5.238 5.366     5.366  7233500   Tesla 2011
...

定位错误

错误信息

Error in checkForRemoteErrors(val) : 
  one node produced an error: 
  In argument: `Company == "Tesla"`.
Caused by error:
! object 'Company' not found

程序员在屏幕上查看一个醒目的红色感叹号。

定位错误

filterCSV <- function (filepath) {

  # Read CSV
  df <- read.csv(filepath)

  # Filter data
  df <- df %>%
    dplyr::filter(Company == "Tesla")

  # Write to back to same path
  write.csv(df, filepath)
}

filterCSV_debug <- function (filepath) {

  df <- read.csv(filepath)


  print(

    # Paste file path and column names
    paste(filepath, ":",

          # Collapse column names into one string
          paste0(colnames(df), collapse = ","))

  )


  df <- df %>% 
    dplyr::filter(Company == "Tesla")

  write.csv(df, filepath)
}

定位错误

cl <- makeCluster(4)
clusterEvalQ(cl, library(dplyr))

dummy <- parLapply(cl, file_list, filterCSV_debug)
stopCluster(cl)

Error in checkForRemoteErrors(val) : 
  one node produced an error: ℹ In argument: `Company == "Microsoft"`.
Caused by error:
! object 'Company' not found

定位错误

cl <- makeCluster(4, outfile = "log.txt") # Log print messages into "log.txt"

clusterEvalQ(cl, library(dplyr))

parLapply(cl, file_list, filterCSV_debug)
stopCluster(cl)

Error in checkForRemoteErrors(val) : 
  one node produced an error: ℹ In argument: `Company == "Tesla"`.
Caused by error:
! object 'Company' not found

检查日志

一段文本显示 CSV 文件路径及其对应的列名。2017 年数据的文件路径被高亮，缺少名为"Company"的列。

使用 foreach 调试

cl <- makeCluster(4,
                  # Supply a text file name to log print messages
                  outfile = "log.txt")

registerDoParallel(cl)

foreach(f = file_list,
        .packages = "dplyr") %dopar% {
  filterCSV_debug(f)
}

stopCluster(cl)

furrr 的好处

plan(multisession, workers = 4)
future_map(file_list, filterCSV_debug)
plan(sequential)

furrr 的好处

[1] "./stocks/2011.csv : Date,Open,High,Low,Close,Adj.Close,Volume,Company,Year"
[1] "./stocks/2012.csv : Date,Open,High,Low,Close,Adj.Close,Volume,Company,Year"
[1] "./stocks/2013.csv : Date,Open,High,Low,Close,Adj.Close,Volume,Company,Year"
[1] "./stocks/2014.csv : Date,Open,High,Low,Close,Adj.Close,Volume,Company,Year"
[1] "./stocks/2015.csv : Date,Open,High,Low,Close,Adj.Close,Volume,Company,Year"
[1] "./stocks/2016.csv : Date,Open,High,Low,Close,Adj.Close,Volume,Company,Year"
[1] "./stocks/2017.csv : Date,Open,High,Low,Close,Adj.Close,Volume,Year"
Error in (function (.x, .f, ..., .progress = FALSE)  : 
  ℹ In index: 1.
Caused by error in `dplyr::filter()`:
ℹ In argument: `Company == "Tesla"`.
Caused by error:
! object 'Company' not found

并行调试

什么是调试？

并行读取文件

过滤函数

并行 apply

顺序运行

定位错误

错误信息

定位错误

定位错误

定位错误

检查日志

使用 foreach 调试

furrr 的好处

furrr 的好处

步骤

并行错误的处理

Passons à la pratique !