Support for various universes (and other submit file parameters)

[jhiemstrawisc]

What would be the best way to pass additional additional values to the underlying submit file you build using HTCondor's python bindings? For example, I have a snakemake workflow I'd like to enable on HTCondor's flagship OSPool. This is a minimal submit file I'm able to run by hand:

universe = container
container_image = docker://jhiemstra/spras:with-uid-v4

# Specify names for log/stdout/stderr files generated by HTCondor
log = spras_$(Cluster).log
output = spras_$(Cluster).out
error = spras_$(Cluster).err

# Specify the script to run inside the container. This is simply a wrapper on the Snakefile
executable = spras.sh

# Handle transferring required inputs/outputs
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
transfer_input_files = example_config.yaml, ../../input, ../../Snakefile, spras.sh
transfer_output_files = output

# System specifications.
request_cpus = 4
request_memory = 4GB
request_disk = 16GB

# Only run on nodes with Singularity installed
requirements = (HAS_SINGULARITY == True)

queue 1

Right now this sends my entire snakemake workflow to a single execution point and runs there, but I'd like to send each snakemake job to its own execution point (I'll ask about moving away from shared filesystems in another issue).

In particular, I'm wondering what the canonical snakemake way to tell the executor what universe and container image it should use when constructing the submit_dict. One could continue to add more and more keys to check against in __init__.py, but condor submit files support so many attributes that this could spiral quickly...

Pardon if this is a naive question, but is there a sense of the best path forward here?