Skip to content

Grid support

danejo3 edited this page Dec 2, 2024 · 4 revisions

DRMAA

To use DRMAA with YEAT, install the DRMAA Python bindings, set up the necessary environment variables, and append grid flags to the YEAT command.

Note: This has been tested only with SGE.

# Install DRMAA python bindings
conda install drmaa

# Set environment variables
export DRMAA_LIBRARY_PATH=/usr/lib/libdrmaa.so.1.0
export SGE_ROOT=/path/to/qsub/bin
export SGE_CELL=default

SLURM

To use SLURM with YEAT, set the required environment variables and append the grid flags.

# Set environment variables
export NFSTMP=$HOME/gridtest
export LD_LIBRARY_PATH=/lib

Grid flags

grid configuration:
  --grid [GRID]         process input in batches using parallel processing on a grid. By default, if `--grid` is invoked with no following arguments, DRMAA
                        will be used to configure jobs on the grid. However, if the scheduler being used is SLURM, users must provide `slurm` as a following
                        argument to `--grid`
  --grid-limit N        limit on the number of concurrent jobs to submit to the grid scheduler; by default, N=1024
  --grid-args A         additional arguments passed to the scheduler to configure grid execution; " -V " is passed by default, or " -V -pe threads <T> "
                        ("sbatch -c <T> " if using SLURM) if --threads is set; this can be used for example to configure grid queue or priority, e.g., " -q
                        largemem -p -1000 " ("sbatch -p largemem --priority -1000 "); note that when overriding the defaults, the user must explicitly add
                        the " -V " ("sbatch") and threads configuration if those are still desired
Clone this wiki locally