SlurmScale is external library for delivering Slurm Elastic Computing.
SlurmScale is a library written in Python that can be used to supply elastic
Slurm cluster scaling. Slurm comes with a built-in support for elastic
computing but it doesn't
appear to have seen any development or support for years so it's not really
functioning as expected. For example, if a CLOUD type Slurm node goes rogue,
[there is no way to remove the node from a
cluster](https://groups.google.com/forum/#!msg/slurm-
devel/QrVL4_Qc3uA/sDU6GeKrCwAJ;context-place=forum/slurm-devel). We also had an
issue with Ansible playbook threads not exiting when called by the provisioning
step. In response, here is a library that can be used to manage the dynamic
portion of a Slurm cluster.
The library can be used for basic Slurm cluster operations, such as listing available cluster nodes, jobs and partitions, including the ability to disable or enable them. Most notably, it can be used to add and remove nodes from a cluster. The feature for adding a node consists or two steps: (1) provisioning step, and (2) configuring step. The provisioning step creates a new instance on a cloud provider while the configuration step integrates it into the cluster. The implementation for these steps is pluggable accounting for flexibility.
The library is intended to be installed on a head node of a Slurm cluster. To install, clone the repository and run the following commands. Note that, for the time being, these instructions are tailored for use with Galaxy Project's infrastructure playbook. Take a look at the [README](https://github.com/ afgane/infrastructure-playbook/blob/custom_elastic/jetstreamiuenv/README.md) in that playbook for instructions on how to modify the playbook configs so it runs as expected. Also note that PySlurm must be installed on the system before this library will install.
sudo -s
# If used w/o infra playbook: mkdir -p /opt/slurm_cloud_provision/
cd /opt/slurm_cloud_provision/
# If used w/o infra playbook: virtualenv .venv && source .venv/bin/activate
source bin/activate
# pip install -U virtualenv packaging appdirs setuptools
git clone https://github.com/afgane/slurmscale.git
cd slurmscale && python setup.py install
pytzmay fail to install as part of the overall setup step. Installing it explicitly seems to work though:pip install -U pytz
Once installed, check the configuration settings in slurmscale.ini.
This configuration file can be placed in the root dir of the
library, in the user's home directory, or in /etc/slurmscale.ini.
cp slurmscale.ini.sample slurmscale.ini
vi slurmscale.ini
Internally, the library uses CloudBridge
library for cross-cloud compatibility so a
CloudBridge configuration file (~/.cloudbridge) is required in addition to
the SlurmScale config file. Galaxy infrastructure playbook will automatically
create this file. If you're not using that playbook, take a look at the
CloudBridge documentation for more details; here is an example of such
file for OpenStack:
[openstack]
os_username: username
os_password: password
os_auth_url: auth url
os_user_domain_name: user domain name
os_project_domain_name: project domain name
os_project_name: project name
In order to configure a provisioned instance, you'll need to supply instance
configuration software. Again, if using the Galaxy infrastructure playbook, it
will place the necessary playbook. If the tool used for provisioning is not
already linked/configured within the library, you'll also need to make
appropriate changes so the library knows how to invoke the software; take a
look at util/provision_manager.py and util/config_manager.py.
Below is an example of usage. Each library method has a docstring with usage details so refer to those for the details.
import slurmscale.jobs
import slurmscale.nodes
import slurmscale.partitions
jobs = slurmscale.jobs.Jobs()
jobs.list()
nodes = slurmscale.nodes.Nodes()
nodes.list()
partitions = slurmscale.partitions.Partitions()
partitions.list()
node = nodes.add()
node.state
node.enable()
node.state
nodes.remove(node)
Library logging can be configured through logging.yaml file in the library's
root folder. By default (during the library's early days), the library will log
everything from the DEBUG level messages to the console while the log file
will only capture WARNING messages. The default log file location is the
library's root folder, called slurmscale.log. You can modify the the file
path and overall debug level in the appropriate handler, or add a logger for a
specific module if you'd like more logging granularity.