add limit on maximum number of running jobs per user

[garlick]

As mentioned in #4309, we need a way to limit the number of running jobs per user, such that when the limit is exceeded, the job is held until a slot becomes available.

Note: this is a distinct problem from #4429 which would reject jobs at submission time if they would cause the user to exceed the user limit on active jobs.

[garlick]

As mentioned in #4429, flux-accounting already implements this. @cmoussa1, would you have a link to issues/prs over in flux-accounting?

Are the jobs "held" some way until they can run as discussed in #4309 or are the rejected at submission time?

[cmoussa1]

flux-framework/flux-accounting#131, flux-framework/flux-accounting#177, and flux-framework/flux-accounting#202 were PR's that all added support for enforcing a max running jobs limit.

Once a user has hit their max running jobs limit, subsequently submitted jobs are held (by adding a dependency to it) until a currently running job has completed and transitioned to job.state.inactive, where a currently held job has its dependency removed and can transition to RUN state.

EDIT: jobs are only rejected at submission time once a user has hit their max active jobs limit.

[garlick]

Great! Thank you! On the tracker I will substitute 131 (active job limit) and 202 (running job limit) and we'll call those tasks done! Yay.

[garlick]

I'll close this now. Chris please double check #4309 and make sure that we have the limits functionality represented by the correct flux-accounting references. If there are any outstanding issues with how those limits work let's add them to the tracker.

[cmoussa1]

I've just left a comment in #4309 to reference the relevant flux-accounting PR's that added support for the respective configurable user limits in the multi-factor priority plugin. Let me know if I missed any details or if any further clarification is required!