Become an expert in R — Interactive courses, Cheat Sheets, certificates and more!
Get Started for Free

parallelStart

Parallelization setup for parallelMap.


Description

Defines the underlying parallelization mode for parallelMap(). Also allows to set a “level” of parallelization. Only calls to parallelMap() with a matching level are parallelized. The defaults of all settings are taken from your options, which you can also define in your R profile. For an introductory tutorial and information on the options configuration, please go to the project's github page at https://github.com/mlr-org/parallelMap.

Usage

parallelStart(
  mode,
  cpus,
  socket.hosts,
  bj.resources = list(),
  bt.resources = list(),
  logging,
  storagedir,
  level,
  load.balancing = FALSE,
  show.info,
  suppress.local.errors = FALSE,
  reproducible,
  ...
)

parallelStartLocal(show.info, suppress.local.errors = FALSE, ...)

parallelStartMulticore(
  cpus,
  logging,
  storagedir,
  level,
  load.balancing = FALSE,
  show.info,
  reproducible,
  ...
)

parallelStartSocket(
  cpus,
  socket.hosts,
  logging,
  storagedir,
  level,
  load.balancing = FALSE,
  show.info,
  reproducible,
  ...
)

parallelStartMPI(
  cpus,
  logging,
  storagedir,
  level,
  load.balancing = FALSE,
  show.info,
  reproducible,
  ...
)

parallelStartBatchJobs(
  bj.resources = list(),
  logging,
  storagedir,
  level,
  show.info,
  ...
)

parallelStartBatchtools(
  bt.resources = list(),
  logging,
  storagedir,
  level,
  show.info,
  ...
)

Arguments

mode

(character(1))
Which parallel mode should be used: “local”, “multicore”, “socket”, “mpi”, “BatchJobs”. Default is the option parallelMap.default.mode or, if not set, “local” without parallel execution.

cpus

(integer(1))
Number of used cpus. For local and BatchJobs mode this argument is ignored. For socket mode, this is the number of processes spawned on localhost, if you want processes on multiple machines use socket.hosts. Default is the option parallelMap.default.cpus or, if not set, parallel::detectCores() for multicore mode, max(1, [mpi.universe.size][Rmpi::mpi.universe.size] - 1) for mpi mode and 1 for socket mode.

socket.hosts

character
Only used in socket mode, otherwise ignored. Names of hosts where parallel processes are spawned. Default is the option parallelMap.default.socket.hosts, if this option exists.

bj.resources

list
Resources like walltime for submitting jobs on HPC clusters via BatchJobs. See BatchJobs::submitJobs(). Defaults are taken from your BatchJobs config file.

bt.resources

list
Analog to bj.resources. See batchtools::submitJobs().

logging

(logical(1))
Should slave output be logged to files via sink() under the storagedir? Files are named <iteration_number>.log and put into unique subdirectories named parallelMap_log_<nr> for each subsequent parallelMap() operation. Previous logging directories are removed on parallelStart if logging is enabled. Logging is not supported for local mode, because you will see all output on the master and can also run stuff like traceback() in case of errors. Default is the option parallelMap.default.logging or, if not set, FALSE.

storagedir

(character(1))
Existing directory where log files and intermediate objects for BatchJobs mode are stored. Note that all nodes must have write access to exactly this path. Default is the current working directory.

level

(character(1))
You can set this so only calls to parallelMap() that have exactly the same level are parallelized. Default is the option parallelMap.default.level or, if not set, NA which means all calls to parallelMap() are are potentially parallelized.

load.balancing

(logical(1))
Enables load balancing for multicore, socket and mpi. Set this to TRUE if you have heterogeneous runtimes. Default is FALSE

show.info

(logical(1))
Verbose output on console for all further package calls? Default is the option parallelMap.default.show.info or, if not set, TRUE.

suppress.local.errors

(logical(1))
Should reporting of error messages during function evaluations in local mode be suppressed? Default ist FALSE, i.e. every error message is shown.

reproducible

(logical(1))
Should parallel jobs produce reproducible results when setting a seed? With this option, parallelMap() calls will be reproducible when using set.seed() with the default RNG kind. This is not the case by default when parallelizing in R, since the default RNG kind "Mersenne-Twister" is not honored by parallel processes. Instead RNG kind "L'Ecuyer-CMRG" needs to be used to ensure paralllel reproducibility. Default is the option parallelMap.default.reproducible or, if not set, TRUE.

...

(any)
Optional parameters, for socket mode passed to parallel::makePSOCKcluster(), for mpi mode passed to parallel::makeCluster() and for multicore passed to parallel::mcmapply() (mc.preschedule (overwriting load.balancing), mc.set.seed, mc.silent and mc.cleanup are supported for multicore).

Details

Currently the following modes are supported, which internally dispatch the mapping operation to functions from different parallelization packages:

local

No parallelization with mapply().

multicore

Multicore execution on a single machine with parallel::mclapply().

mpi

Snow MPI cluster on one or multiple machines with parallel::makeCluster() and parallel::clusterMap().

BatchJobs

Parallelization on batch queuing HPC clusters, e.g., Torque, SLURM, etc., with BatchJobs::batchMap().

For BatchJobs mode you need to define a storage directory through the argument storagedir or the option parallelMap.default.storagedir.

Value

Nothing.


parallelMap

Unified Interface to Parallelization Back-Ends

v1.5.0
BSD_2_clause + file LICENSE
Authors
Bernd Bischl [cre, aut], Michel Lang [aut] (<https://orcid.org/0000-0001-9754-0393>), Patrick Schratz [aut] (<https://orcid.org/0000-0003-0748-6624>)
Initial release

We don't support your browser anymore

Please choose more modern alternatives, such as Google Chrome or Mozilla Firefox.