enable system instance to be restarted without affecting running jobs

[garlick]

As we work through failure modes in the system instance, it would be helpful if brokers or broker subtrees could be rebooted without affecting workloads.

[grondo]

Let's use this issue as a tracking issue for getting basic support for restarting brokers without affecting running jobs. If that is ok then I will open some missing individual issues, and link in those and existing issues in a checklist in the initial post.

To get things started, here's a list of items off the top of my head (I'll open issues on some of these and move them to a checklist above) Anyone should feel free to edit and add to this or a checklist above.

• job-manager:
  • support/test reloading jobs in RUN state from KVS
  • notify exec system of jobs in RUN state after restart (partially described in job-exec module needs a rewrite and new protocol #3346)
• exec system
  • job-exec: replace fork/waitpid with a mechanism which allows "rediscovery" of broker subprocesses #3815
  • job-exec module needs a rewrite and new protocol #3346
  • discover locally running jobs or zombies left from previous broker
• job shell
  • support running in offline mode (shell: offline mode with reconnect #3901)
  • support reconnect: re-register shell service endpoint, re-establish subscriptions, etc.
  • shell: make inter-shell services idempotent #3903
• kvs
  • kvs: support mechanism to checkpoint and restore guest namespaces #3811
• other
  • rc: optionally disable flux job cancelall pushed on cleanup stack in rc1
  • handle: support reconnect (handle: need a way to "reconnect" a flux_t handle #3900)
  • config: configure system instance to use sdexec launcher for job shells

[grondo]

It occurs to me we could develop support for this issue in several phases:

1. restart leaf broker in instance of size > 1 and recover jobs
2. restart non rank 0 broker and its subtree and recover jobs
3. restart rank 0 broker (effectively restart the entire instance)

For the job shell, we could first support restart of single shell jobs, then tackle restart of multi-shell jobs.

So for a first step, we should target a restart of a leaf broker running a single shell rank job.