goslmailer

News & Info

Discussion forums have been enabled for this project, feel free to share your thoughts and ideas there.

v2.7.1

• In v2.7.0 dumping of configuration at goslmailer/gobler startup was removed.
• Now it can be selectively enabled with debugconfig: true parameter in config file

v2.7.0

• slack connector
• mattermost connector
  • mattermost section in json sample config
  • mattermost section in toml sample config
  • mattermost bot sample config

v2.5.0

New: In addition to json, now supports TOML configuration files (.toml file suffix)

• goslmailer json sample config
• goslmailer toml sample config
• gobler toml sample config

BREAKING CHANGE: qosmap changed its format from map[int]string to map[string]int. Adopt the goslmailer config file accordingly.

 "qosmap": {              # map of your sites configured QoSes, with their timelimits (seconds), used for hint generation
-   "3600": "RAPID",
-   "28800": "SHORT",
-   "172800": "MEDIUM",
-   "1209600": "LONG"
 "qosmap": {              # map of your sites configured QoSes, with their timelimits (seconds), used for hint generation
+   "RAPID": 3600,
+   "SHORT": 28800,
+   "MEDIUM": 172800,
+   "LONG": 1209600
 }

Drop-in notification delivery solution for slurm that can do:

• message delivery to:
  • discord
  • matrix
  • telegram
  • msteams
  • slack
  • e-mail
  • mattermost
• gathering of job statistics
• generating hints for users on how to tune their job scripts (see examples below)
• templateable messages (readme)
• message spooling and throttling, details here...
• easy to develop extensions for other protocols, more here...

---

Description

Goslmailer (GoSlurmMailer) is a drop-in replacement MailProg for slurm.

With goslmailer configured as as the slurm mailer,

MailProg                = /usr/bin/goslmailer

it provides users with the ability to specify a comma-separated list of receivers [connector:]target in the sbatch --mail-user switch to select where the messages will be sent out (similar to URI scheme).

e.g.

sbatch --mail-type=ALL --mail-user="mailto:useremailA,msteams:usernameB,telegram:NNNNNNN,usernameC"

To support future additional receiver schemes, a connector package has to be developed and its configuration block present in configuration file.

Currently available connectors:

• discord bot --mail-user=discord:<channelId>
• matrix bot --mail-user=matrix:<roomId>
• telegram bot --mail-user=telegram:<chatId>
• mailto --mail-user=mailto:<email-addr>
• msteams webhook --mail-user=msteams:<userId>
• slack web API --mail-user=slack:<userId> OR --mail-user=slack:<channelId>
• mattermost --mail-user=mattermost:channelId

See each connector details below...

If you would like to contribute to this project by developing a new connector, here you can find a heavily annotated connector boilerplate (fully functional) to help you get started.

---

Building and installing

Option 1. Download latest precompiled binaries here

Unpack, follow instructions

Option 2. Build

Quick version, without end to end testing

git clone https://github.com/CLIP-HPC/goslmailer.git
make test
make build
make install

Slightly more involved, with end to end testing:

Prerequisites:

1. generated RSA keypair (passwordless) (ssh-keygen -t rsa)
2. ssh $USER@localhost must work without password

Known caveats:

• redhat/centos: must have lsb_release binary installed, package: redhat-lsb-core
• ubuntu 22: set enable-bracketed-paste off present in ~/.inputrc
• maybe/maybe not, depends if you see failed tests: export TERM=dumb in ~/.bashrc :)
• in case of the following error:

run_goslmailer: validation *exec.RunRequest failed: target was empty at exec.run  at workflow.run 
run_loop_over_tests_casestest_00: run_goslmailer: validation *exec.RunRequest failed: target was empty at exec.run  at workflow.run 

try:

mkdir $HOME/.secret
ssh-keygen -b 1024 -t rsa -f id_rsa -P "" -f $HOME/.secret/id_rsa
touch ~/.ssh/authorized_keys
cat $HOME/.secret/id_rsa.pub >>  ~/.ssh/authorized_keys 

# downloads endly binary and runs endly tests
make 

Install

goslmailer

• place binary in a path to your liking
• place goslmailer.conf here: /etc/slurm/goslmailer.conf (default path)
  • OR: anywhere else, but then run the binary with GOSLMAILER_CONF=/path/to/gosl.conf in environment
• point slurm MailProg to the binary

gobler

• place binary in a path to your liking
• place gobler.conf in a path to your liking
• start the service (with -c switch pointing to config file)

tgslurmbot

• place binary in a path to your liking
• place tgslurmbot.conf in a path to your liking
  • config file has the same format as goslmailer , so you can use the same one (other connectors configs are not needed)
• start the service (with -c switch pointing to config file)

matrixslurmbot

• place binary in a path to your liking
• place matrixslurmbot.conf in a path to your liking
  • config file has the same format as goslmailer, so you can use the same one (other connectors configs are not needed)
• start the service (with -c switch pointing to config file)

discoslurmbot

• place binary in a path to your liking
• place discoslurmbot.conf in a path to your liking
  • config file has the same format as goslmailer, so you can use the same one (other connectors configs are not needed)
• start the service (with -c switch pointing to config file)

---

Spooling and throttling of messages - gobler service

In high-throughput clusters or in situations where job/message spikes are common, it might not be advisable to try to send all of the incoming messages as they arrive. For these environments goslmailer can be configured to spool messages from certain connectors on disk, to be later processed by the gobler service.

gobler is a daemon program that can be configured to monitor specific spool directories for messages, process them and send out using the same connectors as goslmailer.

On startup, gobler reads its config file and spins-up a connector monitor for each configured spool directory.

connector monitor in turn spins up 3 goroutines: monitor, picker and numSenders x sender.

• monitor :

  • every monitorT seconds (or milliseconds) scans the spoolDir for new messages and sends them to the picker

• picker :

  • on receipt of new messages performs trimming of excessive messages, limiting the number of users messages in the system to maxMsgPU
  • every pickerT seconds (or milliseconds) picks the next message to be delivered and sends it to the sender (ordering by time of arrival)

• sender :

  • numSenders goroutines are waiting for messages from the picker and try to deliver them. In case of failure, messages are returned to the picker for a later retry

Artistic sketch of the system described above

---

Connectors

connector	spooling/throttling capable (gobler)
mattermost	yes
discord	yes
matrix	no
telegram	yes
msteams	yes
mailto	no
slack	yes

default connector

Specifies which receiver scheme is the default one, in case when user didn't specify --mail-user and slurm sent a bare username.

"defaultconnector": "msteams"

---

mailto connector

Mailto covers for original slurm e-mail sending functionality, plus a little bit more. With connector parameters, you can:

• specify your e-mail client (ex slurm: MailProg, e.g. /usr/bin/mutt)
• template mail client command line (e.g. custom subject line)
• template message body
• allowList the recipients

known caveats

• HTML e-mail and mutt
  • to make sure mutt sends HTML email with proper content-type header, add following lines to /etc/Muttrc.local

# Local configuration for Mutt.
set content_type="text/html"

See annotated configuration example

---

mattermost connector

Prerequisites for the mattermost connector:

1. Create mattermost bot, instructions here.
2. OPTIONAL/CONVENIENT: Start the bot daemon service mattermostbot (example config file).
   • This is used to deliver --mail-user=mattermost:chatId messages, can also be looked up by the user in the Info window
3. If the bot is running, it will wake up on the configured triggerString and send the user a private message with slurm job submission instructions
4. Put mattermost block in goslmailer.conf (and optionally gobler.conf if spooling is used)

Additional helpful sources:

• message markdown

---

discord connector

Prerequisites for the discord connector:

1. a discord bot must be created and
2. the bot daemon service discoslurmbot must be running (example config file).
3. once the bot is running, it will wake up on the configured triggerString and send the user a private message with slurm job submission instructions

Discord Bot setup

1. User settings -> Advanced -> Developer mode ON
2. Discord developer portal -> New Application -> Fill out BotName
3. Once the application is saved, select Bot from left menu -> Add Bot -> message: "A wild bot has appeared!"
4. Left menu: OAuth2 -> Copy Client ID
5. Modify this url with the Client ID from 4. and open in browser: https://discord.com/api/oauth2/authorize?client_id=<CLIENT-ID>&permissions=8&scope=bot
6. "An external application BotName wants to access your Discord Account" message -> Select server -> Continue
7. Grant Administrator permissions -> yes/no/maybe ? -> Authorize
8. Discord developer portal -> Select BotName -> Bot menu -> Reset Token -> Copy and Save, to be used in discoslurmbot.conf

Or follow this tutorial

• discord bot and packaged developed using discordgo and with the help of Discord Gophers

---

telegram connector

Sends 1on1 or group chat messages about jobs via telegram messenger app

Prerequisites for the telegram connector:

1. a telegram bot must be created and
2. the bot daemon service tgslumbot must be running.

Site admins can create a telegram bot by messaging botfather.

Once the bot is created, you will receive a bot token. Place the bot token in the goslmailer/gobler config file in the telegram connector section (see example below).

Start the tgslurmbot binary that serves as the bot.

When the chat/group chat with the bot is initiated and/or the bot receives a /start command, he will reply with a chat-specific --mail-user=telegram:nnn message which the user can use in his slurm job scripts to get the job messages.

See annotated configuration example

---

matrix connector

Sends messages to user-defined Matrix rooms.

The Matrix connector can send messages through a pre-defined user that you manually join into channels. In addition, you can setup matrixslurmbot to control this pre-defined user and have him join rooms upon invitation.

The first thing you need is to add the proper parameters to the connectors/matrix section of the config file.

To obtain the access token, you can run this command using curl:

curl -XPOST -d '{"type":"m.login.password", "user":"@myuser:matrix.org", "password":"mypassword"}' "https://matrix.org/_matrix/client/r0/login"

Which should get you a response akin to this one, containing the token:

{
    "user_id": "@myuser:matrix.org",
    "access_token": "syt_dGRpZG9ib3QXXXXXXXEyQMBEmvOVp_10Jm93",
    "home_server": "matrix.org",
    "device_id": "DMIIOMEYBK",
    "well_known": {
        "m.homeserver": {
            "base_url": "https://matrix-client.matrix.org/"
        }
    }
}

Once you've done this, goslmailer will be able to send messages to any channel the configured user is a member of with a command like this:

sbatch --mail-type=ALL --mail-user='matrix:!VEbreVXXXkmLWjeGPi:matrix.org' --wrap "ls"

Where !VEbreVXXXkmLWjeGPi:matrix.org is the ID of the channel you want to send the message to. You can obtain this ID from the Settings>Advanced section of the room.

Using the bot

Instead of manually having the pre-defined user join certain rooms, you can use matrixslurmbot to control this user and have it join rooms automatically upon invitation. For this you just need to start the matrixslurmbot, pointing it to a config file identical to the one before using (you could strip it down to contain only the 'matrix' section).

While the bot is running, any user can invite it to join a channel (e.g. /invite @mybotuser:matrix.org). The bot will join the channel and post the --mail-user:... argument to be used when submitting jobs in order to receive a message in that channel.

In the event that the pre-defined user is alone in a channel (i.e. all normal users left), the bot will make the pre-defined user leave and forget the channel.

---

msteams connector

Sends a message to a preconfigured ms teams channel webhook.

Since MS Teams does not provide with the option to send messages to users directly, only to channel webhooks, we have devised a way using MS Power Automate framework to pick up messages from this one configured sink channel and deliver them via private 1on1 chats to the recipient user.

Users listed in the --mail-user=msteams:userA,msteams:userB will be sent as adaptive card mention entity. A MS Power Automate workflow monitors the configured sink channel, parses the received adaptive card jsons, locates the mention entity and delivers to it the copy of the message via private chat.

See annotated configuration example

---

slack connector

Sends messages to direct messages or channels in Slack using its Web API and slack-go.

Prerequisite for the slack connector:

1. A Slack App must be created

Slack App setup

1. Go to the Slack App portal. Give your app a name (such as "SlurmBot"), and choose the workspace you want to use the app in. You must be allowed to add apps to this workspace.
2. Click on OAuth & Permissions on the sidebar. Scroll down to Scopes and click on Add an OAuth Scope. Add chat:write and im:write.
3. Scroll back up to OAuth Tokens for Your Workspace and copy the Bot User OAuth Token. It should look something like xoxb-5555555555555-5555555555555-XXXXXXXXXXXXXXXXXXXXXXX. This will be used in your goslmailer config (goslmailer.conf or goslmailer.toml), and gobler config if used.

Notes

The bot will be able to send to any channel it's in. Make sure it is not added to channels it shouldn't send messages in (such as an announcments channel). You can add the bot to a channel by pinging it (send a message with just @SlurmBot, for example) and then click on Add to Channel.

• slack-go used in this connector.

---

Gotchas

msteams

• using adaptive card schema version 1.5 doesn't work with our adaptive card, check if some element changed in designer
  • tested: 1.0, 1.2 - work

references

msteams

• Adaptive cards
• Adaptive cards - Designer
• Rate limiting for connectors
• Bot optimizing for rate limiting