-
Notifications
You must be signed in to change notification settings - Fork 0
Grid support
danejo3 edited this page Dec 2, 2024
·
4 revisions
To use DRMAA with YEAT, install the DRMAA Python bindings, set up the necessary environment variables, and append grid flags to the YEAT command.
Note: This has been tested only with SGE.
# Install DRMAA python bindings
conda install drmaa
# Set environment variables
export DRMAA_LIBRARY_PATH=/usr/lib/libdrmaa.so.1.0
export SGE_ROOT=/path/to/qsub/bin
export SGE_CELL=default
To use SLURM with YEAT, set the required environment variables and append the grid flags.
# Set environment variables
export NFSTMP=$HOME/gridtest
export LD_LIBRARY_PATH=/lib
grid configuration:
--grid [GRID] process input in batches using parallel processing on a grid. By default, if `--grid` is invoked with no following arguments, DRMAA
will be used to configure jobs on the grid. However, if the scheduler being used is SLURM, users must provide `slurm` as a following
argument to `--grid`
--grid-limit N limit on the number of concurrent jobs to submit to the grid scheduler; by default, N=1024
--grid-args A additional arguments passed to the scheduler to configure grid execution; " -V " is passed by default, or " -V -pe threads <T> "
("sbatch -c <T> " if using SLURM) if --threads is set; this can be used for example to configure grid queue or priority, e.g., " -q
largemem -p -1000 " ("sbatch -p largemem --priority -1000 "); note that when overriding the defaults, the user must explicitly add
the " -V " ("sbatch") and threads configuration if those are still desired