list executor special options

[cmeesters]

Hi,

when using special executors, these come with additional command line flags, e.g. --slurm-requeue for the slurm executor plugin. These are pretty important to tune the execution, can be placed in a profile/workflow profile and their number will grow over time. It would be nice if a futher version of snk could support their listing under something like Executor Specific Options or similar.

Cheers
Christian

[Wytamma]

Interesting! Ideally I want to keep the snk CLI minimal and only include flags for the specific workflow instance. Snakemake is a powerful workflow engine and snk should hide most of that complexity behind the CLI. That being said I can see how the options would be useful in the help if you install a specific executor with the workflow (the expectation being that you will likely be using that executor).

I think it would be difficult to automatically extract the special executor flags into the Snk CLI. We'd have to introspect the executor and find the ExecutorSettings. This is what snakemake does internally but would require tighter coupling with snakemake to get it working in snk. It would also be easier if we dropped support for snakemake < v8 as executors are a new feature.

An easy option would to be define these in the snk.yaml file. You could then add them to the snk install command and set them at install time. However that'd tie the workflow to specific executors, so you might end up executor specific CLI e.g. workflow-slurm would have a snk.yaml file that explicitly surfaces the slurm-executor settings from the snakemake cli e.g.

cli:
  slurm-requeue:
    from_snakemake: true
-- or --
from_snakemake: ["--slurm-requeue"]

the from_snakemake would tell the CLI to show the option in the help and forward it to snakemake. This would make the CLI more configurable as a whole as there's no way to change the default options currently.

I'll leave this issue open while I think of a solution.