job-manager: add support for housekeeping scripts with partial release of resources #5818
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
garlick
force-pushed
the
housekeeping
branch
3 times, most recently
from
242e109 to
9713a67
Compare
|
rebased on current master |
|
Possibly this is going too far, but to reuse as much tooling as possible, it would be nice if housekeeping "jobs" appeared as a real jobs that you could view with I wonder if we could revive some of the "fastrun" experiment e.g. 9f2a338 to allow the job manager to start instance owner jobs on its own. Then start one of these jobs for housekeeping each time a regular job completes. Bypass the exec system in favor of the direct, stripped down approach proposed here (no shell), and implement partial release using and it would be obvious which nodes are stuck, when that happens. The eventlog could provide a nice record of how nodes behaved. |
|
That's a cool idea! While there would be some interesting benefits, I did consider some issues: Would we end up in the same place because jobs don't currently support partial release? I.e. we'd have to solve partial release and if we did that we could just use traditional epilogs. Also, I wonder what would actually be in the eventlog for these jobs, since there is no scheduling and no job shell? Would it just be a The housekeeping jobs would have to be special cased to avoid running the job prolog and also spawning housekeeping work themselves. Not to say these issues can't be overcome, but I thought I'd bring them up. |
|
Hmm, great points to ponder. I might be trivializing, but I thought partial release from the housekeeping exec to the scheduler could be implemented as it is here, except we'd support I guess that implies some sort of "exec bypass" in the job manager. Perhaps in the spirit of prototyping I could try to tack something on here as a proof of concept if the idea isn't too outlandish. |
|
As a simpler, though less interesting, alternative, we could add a new resource "state" like The job manager could just keep the set of resources in this state and send that set along in the The one thing missing here that's really nice in using the job abstraction (or even the traditional approach of keeping jobs in CLEANUP state) is that there's no tracking of how long resources have been doing housekeeping/maintenance. Probably would need some way to provide that. 🤔 |
It does not seem too outlandish for a prototype. My main worry is having two classes of jobs, and the fallout this could cause among all the tools. However, I do see why tracking these things as jobs is compelling. |
This might be good to do anyway since there's interest in allowing jobs to release resources while still running. We'll have to think through how tools and the job record reflect a changing resource set. |
|
Mentioning @ryanday36. Ryan, this is a new epilog like capability that runs after each job but does not prevent the job from completing. It also supports "partial release" whereby any stragglers are returned to the scheduler independently. The main topic of discussion is tooling. How do you tell that housekeeping is still running on node(s) and for how long? Some other aspects are:
|
|
This seems like an interesting approach. As far as tooling, if you do list the housekeeping jobs are "real" jobs, I like that the they are listed as belonging to the I think that using I agree that yes, it would be good to track nodes that have unfinished housekeeping. My last thought is a sort of meta-concern. It seems like whenever we talk about the current prolog / epilog it's described as something that you're not quite happy with. Would this be further cementing that implementation, or would it be separating concerns in a way that would allow you to make changes to the current prolog / epilog more easily? |
My worry is that unanticipated errors from a script or set or scripts run in this way will be impossible to diagnose after the fact. You can easily code your housekeeping script to do try_something || flux logger "something failed"Are we going to suggest authors do that on every line? What if your python script throws an unexpected exception, are we going to suggest cleanup scripts are wrapped in a large If we represent these things as jobs, we have a nice way to capture output from jobs, so maybe we could just do that? Is the worry that it is just too much work, or that it will double the KVS usage since there will be one housekeeping job per actual job? |
|
Good points. I guess I was thinking of the runner script that they have now that could log errors from the scripts that it runs, but it's actually easy enough to capture output if we need it. The downside is if we capture it in the flux log like we do now, it can be a source of log noise that pushes out the interesting stuff. I think its very easy to add. |
|
If these housekeeping scripts are emulated like jobs could we just log the errors in an output eventlog? |
The problem with the current way the per-node epilog is managed is that it uses a facility that was designed for things that need to run once in a central location after each job to remotely execute the epilog using a specialized script. This was meant to be a stopgap until we could distribute the job execution system, which would then be able to manage the epilog scripts directly from each broker. However, in my opinion, the solution proposed here is superior to even what we had planned. The benefits I see are:
The only drawback I can see is perhaps more muddled reporting -- it isn't clear what user job spawned a housekeeping job (if that is the way it is going to be reported). But that could easily be queried, or could even be an annotation come to think of it. Maybe that is actually a benefit. Also, reporting housekeeping work as jobs will (I assume?) report the nodes as "allocated" instead of some other state, and when queries are made for jobs there will always be some housekeeping jobs reported that would likely have to be filtered out. |
|
Cool. Thanks for the discussion on the benefits @grondo. That makes me feel better about it. I do think that it would be good to be able to correlate housekeeping jobs with the user job that launched them. It's good to be able to find out if one user is consistently managing to leave nodes in bad states. I'd actually be interested in being able to track the time spent in housekeeping separately from the time running the user jobs. It could be a good tool for keeping run_ansible and other epilog things from getting out of hand. That said, for most reporting, we'd probably lump them in as 'allocated'. I believe that's how time spent in 'completing' currently gets counted in Slurm. Lastly, good points on the logging discussion. Things like run_ansible do generate a ton of output that would probably be good to keep out of the flux log. It would be good to be able to get to it if the housekeeping job fails though, so a job specific output event log that could be referenced if the job fails seems like it would be useful. |
|
Rebased on current master with no other changes. |
|
|
||
| if (!(hk = calloc (1, sizeof (*hk)))) | ||
| return NULL; | ||
| hk->ctx = ctx; |
Check warning
Code scanning / CodeQL
Local variable address stored in non-local memory Warning
|
Changes in that last push were
Per offline discussion, next steps are
|
|
Trying on my test cluster. I did get the following from the bash completions: otherwise looks good so far! |
garlick
changed the title
garlick
changed the title
|
Thanks for testing! I thought I'd fixed that one, sorry! I'll fix it up when I get back to a computer |
|
Ok, fixed that bash completion error (it was a mistake in fixing up a conflict) |
|
Thanks! I just pushed a couple of typos (mine!) |
|
fixup commits LGTM! (though |
|
Oops! Ok I'll fix that. OK to squash the fixups and rebase on master? |
|
OK! 🎺 🎺 |
Problem: the bulk-exec API in job-exec is useful elsewhere, but it is local to the job-exec module. Move it into the libsubprocess directory where it can be more easily reused within flux-core (it is not added to the public API). Update job-exec.
Problem: free requests may not take place in the context of the job once housekeeping is in place, but t2212-job-manager-plugins.t uses the debug.free-request event as an indication that the job debug flag could be set. Use debug.alloc-request in the test instead.
Problem: jobs get stuck in CLEANUP state while long epilog scripts run, causing sadness and idling resources. Introduce a new type of epilog script called "housekeeping" that runs after the job. Instead of freeing resources directly to the scheduler, jobs free resources to housekeeping, post their free event, and may reach INACTIVE state. Meanwhile, housekeeping can run a script on the allocated resources and return the resources to the scheduler when complete. The resources are still allocated to the job as far as the scheduler is concerned while housekeeping runs. However since the job has transitioned to INACTIVE, the flux-accounting plugin will decrement the running job count for the user and stop billing the user for the resources. 'flux resource list' utility shows the resources as allocated. By default, resources are released to the scheduler only after all ranks complete housekeeping, as before. However, if configured, resources can be freed to the scheduler immediately as they complete housekeeping on each execution target, or a timer can be started on completion of the first target, and when the timer expires, all the targets that have completed thus far are freed in one go. Following that, resources are freed to the scheduler immediately as they complete. This works with sched-simple without changes, with the exception that the hello protocol does not currently support partial release so, as noted in the code, housekeeping and a new job could overlap when the scheduler is reloaded on a live system. Some RFC 27 work is needed to resolve ths. The Fluxion scheduler does not currently support partial release (flux-framework/flux-sched#1151). But as discussed over there, the combination of receiving an R fragment and a jobid in the free request should be sufficient to get that working.
Problem: housekeeping scripts should be run in a systemd cgroup for reliable termination, logging to the local systemd journal, and debugging using well known systemd tools. Add a systemd "oneshot" service unit for housekeeping, templated by jobid and corresponding script that can be configured as an imp run target. This is similar to what was proposed for prolog/epilog except if the script fails, the node is drained.
Problem: bulk_exec does not offer an interface to kill a subset of the running subprocesses. Add a ranks parameter to bulk_exec_kill() and bulk_exec_imp_kill(). Set it to NULL to target all subprocesses like before. Update test and users.
Problem: There is interface to query information about the housekeeping subsystem. Add a new flux-housekeeping(1) command, which can be used to query or terminate housekeeping tasks.
Problem: The flux-housekeeping(1) command doesn't have any bash tab completions. Add them.
Problem: there is no test coverage for job-manager housekeeping. Add a sharness script.
Problem: There is no manual for flux-housekeeping(1). Add a simple man page for flux-housekeeping(1).
Problem: The job-manager.housekeeping config table is not documented. Document it in flux-config-job-manager(5).
Problem: the text to explain expectations for experimental features must be repeated in documentation for each one which is extra work for both the author and the reader. Add common/experimental.rst which can be included from man pages that discuss experimental features. Like common/resource.rst, the idea is to make it a standalone section which can then be referenced from the feature documentation. This should improve consistency and make it easier to document these features and interfaces. Fixes flux-framework#6066
Problem: the EXPERIMENTAL section refers to the project issue tracker but man pages do not contain the URL. Add it.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5818 +/- ##
==========================================
- Coverage 83.34% 83.34% -0.01%
==========================================
Files 519 521 +2
Lines 83819 84305 +486
==========================================
+ Hits 69862 70261 +399
- Misses 13957 14044 +87
|
This is a prototype of an idea @grondo and I were discussing last week after someone was complaining about a large job stuck in CLEANUP state while the epilog ran (reminding us with a pang of guilt that the lack of "partial release" support is going to be an ongoing source of sadness). This introduces a new type of epilog script called "housekeeping" that runs after the one we have now. There are two main features here
sched-simple supports (2) above. Fluxion does not, yet. See flux-framework/flux-sched#1151.
The idea was that maybe we could move some things that take a long time (like ansible updates) and that are not job-specific, from the epilog to housekeeping. In addition, working on this will remove some barriers to implementing partial release of resources from jobs, which is more involved, but the ultimate goal.
One loose end is the RFC 27 hello request. Currently an R fragment cannot be passed back that way. For now, if a job's allocation is partially freed when the scheduler is reloaded, it is logged at LOG_ERR and not included in the list of pre-allocated jobs. We accept that we might schedule a job on a node before housekeeping is finished when the scheduler is reloaded (hopefully rare). More design work required for RFC 27.
Another is tooling.
flux resource listshows resources in housekeeping as allocated. It would be nice to know at a glance which nodes are tied up in housekeeping and maybe provide other information that would add transparency and improve how the admins interact with the machine.This is currently based on #5796 because there was a bit of overlap in the code, but conceptually they are separate features. At the time this PR was posted, the new work is the top 4 commits.I probably jumped the gun going this far with this without getting some feedback on the approach. I'll just say it's only a couple days work and I learned a fair bit about the problem space so we can throw this out and start over if it ends up being off base.