Virtual Galaxy Compute Nodes

Galaxy Europe runs jobs on compute nodes belonging to the bwCloud/de.NBI-cloud. These compute nodes are known as "Virtual Galaxy Compute Nodes" (VGCN).

Virtual Galaxy Compute Nodes boot from VGCN images, which are built off this repository and made available as GitHub releases and on this static site.

Virtual Galaxy Compute Node images

Virtual Galaxy Compute Node images contain everything needed to run Galaxy jobs.

• Apptainer
• CVMFS
• Docker
• NFS
• Pulsar
• Telegraf

Each service can be further configured via cloud-init scripts to meet specific needs. A few setup ideas:

• Like on usegalaxy.eu: one machine as the HTCondor master, the rest as job runners.
• Like usegalaxy.eu's remote clusters: one machine as the HTCondor master and pulsar server, another as NFS file server, the rest as job runners.
• For "Bring Your own Compute" (BYOC) use cases: A single node that acts as HTCondor master, Pulsar server, and job runner.

Important: HTCondor has a new authentication mechanism, which uses passwords and tokens, generated automatically during startup. The easiest and least error-prone way to set this up is during installation.

! This is why we removed HTCondor from public images.

You can easily install it and set up your password by adding the execute installation line from the link above to your cloud-init script. There you can set your own password and host. Both should be identical on all machines you use, the host is the hostname or IP address of your central manager - usually the node that accepts Pulsar requests. An implementation in Pulsar-Infrastructure-Playbook is in progress.

Build instructions

Make sure the following packages are installed on your system.

• Packer >= 1.9.1, < 2
• QEMU >= 6.2, < 9
• Ansible, see requirements.txt

Run Packer to build the images.

packer build \
    -only=qemu.rockylinux-8.6-x86_64,qemu.rockylinux-9.2-x86_64 \
    -var="headless=true" \
    -var='groups=["generic", "workers", "external"]' \
    templates

• -only=qemu.rockylinux-8.6-x86_64,qemu.rockylinux-9.2-x86_64: selects the underlying operating system on which the images will be based. One image will be produced for each item. The argument can be omitted to produce images for all supported operating systems. All builds use the QEMU builder, hence the prefix. Supported operating systems are listed in build.pkr.hcl.
• -var="headless=true": display the screen of the QEMU virtual machines used to build the images by setting this variable to false.
• -var='groups=["generic", "workers", "external"]': Playbooks that the Packer Ansible provisioner will run. VGCN standard images are built with the setting groups=["generic", "workers", "external"]. Add workers-gpu to the list to get the GPU images. Read the comments in variables.pkr.hcl for more details.
• templates: the directory containing the Packer templates.

Once the images are built, they will be available in a new directory called "images".

Running VGCN images

Please see https://github.com/usegalaxy-eu/terraform/ for examples of how to launch and configure this.