Cannot use read.table () inside foreach loop (doSMP)

I am trying to use doSMP / foreach to parallelize some code in R.

I had a huge 2d matrix of genetic data - 10,000 observations (rows) and 3 million variables (columns). I had to break this data into pieces of 1000 variables due to memory problems.

I want to read in each file, do some statistics and write these results to a file. This is easy with a for loop, but I want to use foreach to speed it up. That's what I'm doing:

# load doSMP, foreach, iterators, codetools
require(doSMP)

# files i'm processing
print(filelist <- system("ls matrix1k.*.txt", T))

#initialize processes
w <- startWorkers(2)
registerDoSMP(w)

# for each file, read into memory, do some stuff, write out results.
foreach (i =  1:length(filelist)) %dopar% {
    print(i)
    file <- filelist[i]
    print(file)
    thisfile <- read.table(file,header=T) 
    # here i'll do stuff using that file
    # here i'll write out results of the stuff I do above
}

#stop processes
stopWorkers(w)

But this leads to an error: Error in { : task 2 failed - "cannot open the connection". When I change %dopar%to %do%, there is no problem.

+3
source share
2 answers

, . , , 2 , . Disk IO - ( ), RAID- . IO , .

+1

foreach , .

:

i)

foreach (i =  1:length(filelist), .packages = "rgdal") %dopar% ......

.

2:

ii)

package.vector <- c("package.1","package.2",etc)

foreach (i =  1:length(filelist), .packages = package.vector) %dopar% ......

,

+2

All Articles