Version:	0.12.2
Depends:	R (≥ 3.2.0), parallelly, future (≥ 1.49.0)
Imports:	batchtools (≥ 0.9.16), utils
Suggests:	globals, future.apply, listenv, markdown, R.rsp
VignetteBuilder:	R.rsp
Title:	A Future API for Parallel and Distributed Processing using 'batchtools'
Description:	Implementation of the Future API <doi:10.32614/RJ-2021-048> on top of the 'batchtools' package. This allows you to process futures, as defined by the 'future' package, in parallel out of the box, not only on your local machine or ad-hoc cluster of machines, but also via high-performance compute ('HPC') job schedulers such as 'LSF', 'OpenLava', 'Slurm', 'SGE', and 'TORQUE' / 'PBS', e.g. 'y <- future.apply::future_lapply(files, FUN = process)'.
License:	LGPL-2.1 | LGPL-3 [expanded from: LGPL (≥ 2.1)]
LazyLoad:	TRUE
URL:	https://future.batchtools.futureverse.org, https://github.com/futureverse/future.batchtools
BugReports:	https://github.com/futureverse/future.batchtools/issues
RoxygenNote:	7.3.2
NeedsCompilation:	no
Packaged:	2025-06-06 02:24:53 UTC; henrik
Author:	Henrik Bengtsson [aut, cre, cph]
Maintainer:	Henrik Bengtsson <henrikb@braju.com>
Repository:	CRAN
Date/Publication:	2025-06-06 02:50:02 UTC

future.batchtools: A Future for batchtools

Description

The future.batchtools package implements the Future API on top of batchtools such that futures can be resolved on for instance high-performance compute (HPC) clusters via job schedulers. The Future API is defined by the future package.

Details

To use batchtools futures, load future.batchtools, and select the type of future you wish to use via future::plan().

Author(s)

Maintainer: Henrik Bengtsson henrikb@braju.com (ORCID) [copyright holder]

See Also

Useful links:

• https://future.batchtools.futureverse.org

• https://github.com/futureverse/future.batchtools

• Report bugs at https://github.com/futureverse/future.batchtools/issues

Examples

library(future.batchtools)

## Use local batchtools futures
plan(batchtools_local)

## A global variable
a <- 1

v %<-% {
  b <- 3
  c <- 2
  a * b * c
}

print(v)

plan(batchtools_local)
demo("mandelbrot", package = "future", ask = FALSE)

Temporarily tweaks the resources for the current batchtools strategy

Description

Temporarily tweaks the resources for the current batchtools strategy

Usage

fassignment %resources% tweaks

Arguments

A batchtools future is a future whose value will be resolved via batchtools

Description

A batchtools future is a future whose value will be resolved via batchtools

Usage

BatchtoolsCustomFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  ...
)

BatchtoolsBashFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  ...
)

BatchtoolsFuture(
  expr = NULL,
  envir = parent.frame(),
  substitute = TRUE,
  globals = TRUE,
  packages = NULL,
  label = NULL,
  resources = list(),
  workers = NULL,
  finalize = getOption("future.finalize", TRUE),
  conf.file = findConfFile(),
  cluster.functions = NULL,
  registry = list(),
  ...
)

BatchtoolsMultiprocessFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  ...
)

BatchtoolsMulticoreFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  ...
)

BatchtoolsTemplateFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  ...
)

BatchtoolsLsfFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  ...
)

BatchtoolsOpenLavaFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  ...
)

BatchtoolsSGEFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  ...
)

BatchtoolsSlurmFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  ...
)

BatchtoolsTorqueFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  ...
)

BatchtoolsSSHFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  workers = availableCores(),
  ...
)

BatchtoolsUniprocessFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  ...
)

BatchtoolsLocalFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  ...
)

BatchtoolsInteractiveFuture(
  expr = NULL,
  substitute = TRUE,
  envir = parent.frame(),
  ...
)

Arguments

Value

A BatchtoolsFuture object

FutureError class for errors related to BatchtoolsFuture:s

Description

FutureError class for errors related to BatchtoolsFuture:s

Usage

BatchtoolsFutureError(...)

Arguments

Batchtools futures for custom batchtools configuration

Description

Batchtools futures for custom batchtools configuration

Usage

batchtools_custom(
  expr,
  envir = parent.frame(),
  substitute = TRUE,
  globals = TRUE,
  label = NULL,
  resources = list(),
  workers = NULL,
  conf.file = findConfFile(),
  cluster.functions = NULL,
  registry = list(),
  ...
)

Arguments

Value

An object of class BatchtoolsFuture.

Examples

options(error = function(...) {
  print(traceback())
})

cf <- batchtools::makeClusterFunctionsInteractive(external = TRUE)
print(cf)
str(cf)
plan(batchtools_custom, cluster.functions = cf)
print(plan())
print(nbrOfWorkers())

## Create explicit future
f <- future({
  cat("PID:", Sys.getpid(), "\n")
  42L
})
print(f)
v <- value(f)
print(v)

options(error = NULL)


## Create explicit future
f <- future({
  cat("PID:", Sys.getpid(), "\n")
  42L
})
print(f)
v <- value(f)
print(v)



## Create explicit future
f <- future({
  cat("PID:", Sys.getpid(), "\n")
  42L
})
v <- value(f)
print(v)

batchtools local and interactive futures

Description

A batchtools local future is an synchronous uniprocess future that will be evaluated in a background R session. A batchtools interactive future is an synchronous uniprocess future that will be evaluated in the current R session (and variables will be assigned to the calling environment rather than to a local one). Both types of futures will block until the futures are resolved.

Usage

batchtools_local(..., envir = parent.frame())

Arguments

Details

batchtools local futures rely on the batchtools backend set up by batchtools::makeClusterFunctionsInteractive(external = TRUE) and batchtools interactive futures on the one set up by batchtools::makeClusterFunctionsInteractive(). These are supported by all operating systems.

An alternative to batchtools local futures is to use cluster futures of the future package with a single local background session, i.e. plan(cluster, workers = "localhost").

An alternative to batchtools interactive futures is to use plan(sequential, split = TRUE) futures of the future package.

Value

An object of class BatchtoolsUniprocessFuture.

Examples

## Use local batchtools futures
plan(batchtools_local)

## A global variable
a <- 1

## Create explicit future
f <- future({
  b <- 3
  c <- 2
  a * b * c
})
v <- value(f)
print(v)


## Create implicit future
v %<-% {
  b <- 3
  c <- 2
  a * b * c
}
print(v)

batchtools multicore futures

Description

A batchtools multicore future is an asynchronous multiprocess future that will be evaluated in a background R session.

We highly recommend using future::multisession (sic!) futures of the future package instead of multicore batchtools futures.

Usage

batchtools_multicore(
  expr,
  envir = parent.frame(),
  substitute = TRUE,
  globals = TRUE,
  label = NULL,
  workers = availableCores(constraints = "multicore"),
  registry = list(),
  ...
)

Arguments

Details

batchtools multicore futures rely on the batchtools backend set up by batchtools::makeClusterFunctionsMulticore(). The batchtools multicore backend only works on operating systems supporting the ps command-line tool, e.g. Linux and macOS.

Value

An object of class BatchtoolsMulticoreFuture.

batchtools SSH futures

Description

A batchtools SSH future is an asynchronous multiprocess future that will be evaluated in a background R session.

We highly recommend using future::multisession (sic!) futures of the future package instead of SSH batchtools futures.

Usage

batchtools_ssh(
  expr,
  envir = parent.frame(),
  substitute = TRUE,
  globals = TRUE,
  label = NULL,
  workers = availableWorkers(),
  registry = list(),
  ...
)

Arguments

Details

batchtools SSH futures rely on the batchtools backend set up by batchtools::makeClusterFunctionsSSH(). The batchtools SSH backend only works on operating systems supporting the ssh and ps command-line tool, e.g. Linux and macOS.

Value

An object of class BatchtoolsMulticoreFuture.

Batchtools futures for LSF, OpenLava, SGE, Slurm, TORQUE etc.

Description

Batchtools futures for LSF, OpenLava, SGE, Slurm, TORQUE etc. are asynchronous multiprocess futures that will be evaluated on a compute cluster via a job scheduler.

Usage

batchtools_lsf(
  expr,
  envir = parent.frame(),
  substitute = TRUE,
  globals = TRUE,
  label = NULL,
  template = NULL,
  resources = list(),
  workers = NULL,
  registry = list(),
  ...
)

batchtools_openlava(
  expr,
  envir = parent.frame(),
  substitute = TRUE,
  globals = TRUE,
  label = NULL,
  template = NULL,
  resources = list(),
  workers = NULL,
  registry = list(),
  ...
)

batchtools_sge(
  expr,
  envir = parent.frame(),
  substitute = TRUE,
  globals = TRUE,
  label = NULL,
  template = NULL,
  resources = list(),
  workers = NULL,
  registry = list(),
  ...
)

batchtools_slurm(
  expr,
  envir = parent.frame(),
  substitute = TRUE,
  globals = TRUE,
  label = NULL,
  template = NULL,
  resources = list(),
  workers = NULL,
  registry = list(),
  ...
)

batchtools_torque(
  expr,
  envir = parent.frame(),
  substitute = TRUE,
  globals = TRUE,
  label = NULL,
  template = NULL,
  resources = list(),
  workers = NULL,
  registry = list(),
  ...
)

Arguments

Details

These type of batchtools futures rely on batchtools backends set up using the following batchtools functions:

• batchtools::makeClusterFunctionsLSF() for Load Sharing Facility (LSF)

• batchtools::makeClusterFunctionsOpenLava() for OpenLava

• batchtools::makeClusterFunctionsSGE() for Sun/Oracle Grid Engine (SGE)

• batchtools::makeClusterFunctionsSlurm() for Slurm

• batchtools::makeClusterFunctionsTORQUE() for TORQUE / PBS

Value

An object of class BatchtoolsFuture.

Removes a batchtools future

Description

Removes a batchtools future

Usage

## S3 method for class 'BatchtoolsFuture'
delete(
  future,
  onRunning = c("warning", "error", "skip"),
  onFailure = c("error", "warning", "ignore"),
  onMissing = c("ignore", "warning", "error"),
  times = 10L,
  delta = getOption("future.wait.interval", 1),
  alpha = getOption("future.wait.alpha", 1.01),
  ...
)

Arguments

Value

(invisibly) TRUE if deleted and FALSE otherwise.

Options used for batchtools futures

Description

Below are the R options and environment variables that are used by the future.batchtools package. See future::future.options for additional ones that apply to futures in general.

WARNING: Note that the names and the default values of these options may change in future versions of the package. Please use with care until further notice.

Settings for batchtools futures

future.batchtools.workers:

(a positive numeric or +Inf) The default number of workers available on HPC schedulers with job queues. (Default: 100)

future.batchtools.output:

(logical) If TRUE, batchtools will produce extra output. If FALSE, such output will be disabled by setting batchtools options batchtools.verbose and batchtools.progress to FALSE. (Default: getOption("future.debug", FALSE))

future.batchtools.expiration.tail:

(a positive numeric) When a batchtools job expires, the last few lines will be relayed by batchtools futures to help troubleshooting. This option controls how many lines are displayed. (Default: 48L)

future.cache.path:

(character string) An absolute or relative path specifying the root folder in which batchtools registry folders are stored. This folder needs to be accessible from all hosts ("workers"). Specifically, it must not be a folder that is only local to the machine such as ⁠file.path(tempdir(), ".future"⁠ if an job scheduler on a HPC environment is used. (Default: .future in the current working directory)

future.delete:

(logical) Controls whether or not the future's batchtools registry folder is deleted after the future result has been collected. If TRUE, it is always deleted. If FALSE, it is never deleted. If not set or NULL, the it is deleted, unless running in non-interactive mode and the batchtools job failed or expired, which helps to troubleshoot when running in batch mode. (Default: NULL (not set))

Environment variables that set R options

All of the above R future.batchtools.* options can be set by corresponding environment variable R_FUTURE_BATCHTOOLS_* when the future.batchtools package is loaded. This means that those environment variables must be set before the future.batchtools package is loaded in order to have an effect. For example, if R_FUTURE_BATCHTOOLS_WORKERS="200" is set, then option future.batchtools.workers is set to 200 (numeric).

Examples

# Set an R option:
options(future.cache.path = "/cluster-wide/folder/.future")

Logged output of batchtools future

Description

Logged output of batchtools future

Usage

loggedOutput(...)

## S3 method for class 'BatchtoolsFuture'
loggedError(future, ...)

Arguments

Value

A character vector or a logical scalar.

Gets the number of batchtools workers

Description

Tries to infer the total number of batchtools workers. This is done using various ad-hoc procedures based on code inspection of batchtools itself.

Usage

## S3 method for class 'batchtools'
nbrOfWorkers(evaluator)

Arguments

Value

A number in [1, Inf].

Prints a batchtools future

Description

Prints a batchtools future

Usage

## S3 method for class 'BatchtoolsFuture'
print(x, ...)

Arguments