configure system instance to require every job be a new instance

[grondo]

(I thought we had captured this requirement somewhere, but I'm unable to find it at the moment)

We've talked about somehow enforcing that all jobs submitted to a system instance are new instances of Flux, i.e. they are submitted with jobspecs equivalent to flux mini batch or flux mini alloc. Some of the reasons for this requirement include:

• Launching a new instance for every job directly launched from the system instance offloads much of the KVS and message traffic (e.g. all output is typically directed to a file or interactive pty, though this can always be overridden on the command line or jobspec, PMI for MPI jobs within the new instance are handled by subinstance broker wireup)
• I think at one point we were thinking that sub-instances would be easier to recover from a system instance restart than individual MPI or other jobs, but I'm not sure that is the case anymore.
• I actually thought this matches how we configure our Slurm clusters, but that doesn't seem to be the case (you can run an interactive srun in the batch partition just fine if you are patient enough)

Given that the second two bullets are not really true anymore, I wonder if this is still a requirement. It does seem like the first bullet has enough benefits that we might want to at least consider implementing a solution.

On the other hand, I'm not sure there is a practical way to determine that a submitted jobspec definitely results in a new instance of Flux. So this needs a little more thought.

[dongahn]

It does seem like the first bullet has enough benefits that we might want to at least consider implementing a solution.

Yes, if I remember right, reducing the load at the system instance was the main motivation.

On the other hand, I'm not sure there is a practical way to determine that a submitted jobspect definitely results in a new instance of Flux. So this needs a little more thought.

Can we add a key to attributes section of the jobspec if the flux mini batch or alloc is used and have a job validator plugin for our system instance to validate it only this key exists? I don't even know if it is possible to configure a different validator for the system instance (e.g., TOML. configurable). So take it just as a food for thought.

[grondo]

Can we add a key to attributes section of the jobspec if the flux mini batch or alloc is used and have a job validator plugin for our system instance to validate it only this key exists?

We could, though nothing prevents a user from setting this key themselves, e.g. flux mini submit --cc=1-1024 --setattr=system.mini-batch=true ... so it seems a little kludgy...

[dongahn]

Or maybe the validator can look at the task section and check if that invokes flux broker or has attributes.batch

[grondo]

Ok, so a system instance validator plugin could employ the following couple of checks: If any check is true then the job is validated for submission to the instance:

• if attributes.system.batch is set with a valid script
• if the basename of tasks[0].command[0] is flux, and tasks[0].command[1] is broker or start

I think you are right this should cover 99% of cases. Possibly we should distribute the validator plugin outside of flux-core in case there are some corner cases the plugin could be updated on-the-fly.

Here's a proof of concept plugin:

"""Only allow jobs that launch a new instance of Flux
"""

import errno
from os.path import basename
from flux.job.validator import ValidatorPlugin


class Validator(ValidatorPlugin):
    def validate(self, args):
        if "batch" in args.jobspec["attributes"]["system"]:
            return
        command = args.jobspec["tasks"][0]["command"]
        arg0 = basename(command[0])
        if arg0 == "flux" and command[1] in ["broker", "start"]:
            return
        return (
            errno.EINVAL,
            "Direct job submission is disabled for this instance."
            + " Please use the batch or alloc subcommands of flux-mini(1)",
        )

$ flux mini run hostname
flux-mini: ERROR: [Errno 22] Direct job submission is disabled for this instance. Please use the batch or alloc subcommands of flux-mini(1)
$ flux mini alloc -n1 hostname
fluke108
[detached: session exiting]

[dongahn]

So fast!