Geek asks and answers

Transfer the entire packet to the snow cluster

I am trying to parallelize (using snow::parLapply) some code that depends on a package (i.e. a package other than snow). Objects referenced by the function being called parLapplymust be explicitly passed to the cluster using clusterExport. Is there a way to transfer the entire package to the cluster, instead of explicitly specifying each function (including the internal functions of the package called by user functions!) Q clusterExport?

+5
source share
1 answer

Install the package on all nodes and get a code call library(thePackageYouUse)on all nodes using the available commands, something like

 clusterApply(cl, library(thePackageYouUse))

, parallel, R, - . , , help(clusterApply), boot :

 ## A bootstrapping example, which can be done in many ways:
 clusterEvalQ(cl, {
   ## set up each worker.  Could also use clusterExport()
   library(boot)
   cd4.rg <- function(data, mle) MASS::mvrnorm(nrow(data), mle$m, mle$v)
   cd4.mle <- list(m = colMeans(cd4), v = var(cd4))
   NULL
 })
+6

More articles:


All Articles