job-manager: implement state synchronization (many job)

[garlick]

We need a scalable method for entities like workflow managers, the simulator, and other tools, to be notiifed in bulk of job state transtions. This is distinct from the single job synchronization interface proposed in #1665. The driver for a separate bulk interface is scalability.

One approach would be to publish events like we did in the old system. This has some appeal due to its simplicity, and lack of impact (or at least uniform impact) on the job-manager. To improve scalability we could batch state transitions to reduce the number of event messages, e.g. publish a JSON dictionary mapping states to arrays of jobids entering that state, slightly delayed. We could put an API in front of the event messages, to arrange for a callback for each job, e.g.

void (*bulk_state_callback_f)(flux_jobid_t id, flux_job_state_t state, void *arg);

Some other questions:

• Do we need a way for a late joining tool to "catch up" on historical state transitions?
• Do we need events for all states or could we get by with, say, NEW and CLEANUP?

[grondo]

Does event-based notification imply only instance owners will have access to bulk state synchronization?

We could put an API in front of the event messages, to arrange for a callback for each job, e.g.

Instead of another callback_f in the Flux API, what if the bulk watch API function returned a flux_future_t which could be fulfilled multiple times. A set of flux_job_watch_get(3) functions (or something) could be shared between flux_job_wait(3) and flux_job_wait_all(3)?

[garlick]

Does event-based notification imply only instance owners will have access to bulk state synchronization?

It depends on whether we set the privacy bit on the event. If there is not a good reason to specify that these events are private, I guess I would advocate for making the interface accessible to all. Not much is exposed here that would be sensitive.

Instead of another callback_f in the Flux API, what if the bulk watch API function returned a flux_future_t which could be fulfilled multiple times. A set of flux_job_watch_get(3) functions (or something) could be shared between flux_job_wait(3) and flux_job_wait_all(3)?

Excellent thought! I don't see why we couldn't do that.

[grondo]

It depends on whether we set the privacy bit on the event. If there is not a good reason to specify that these events are private, I guess I would advocate for making the interface accessible to all. Not much is exposed here that would be sensitive.

Ok, great! I had forgotten about the ability to disable privacy of event messages.

[garlick]

One point brought up by @SteVwonder for e.g. the simulator use case:

1. subscribe to events
2. start workload

could be subject to a race if 1) is "fire and forget", e.g. the subscrption is not in place until some later time, resulting in missed job state changes if the workload is immediately started.

I double checked and flux_event_subscribe() should be fully synchronous. That is, it does not return until the subscription is in place.

Here I should reference #1557 which notes that this API call should actually return a future since it hides an RPC, and also that if someone digs down to fix this, they will find there is a "connector operation" implementing it rather than a simple RPC. The connector operation is no longer justified and should probably be replaced with a more straightforward approach.

Anyway, the point above was a good one to consider, but it should not be a problem with the current implementation of event subscriptions.

[grondo]

Not sure, but does the current implementation of flux_job_wait(3) satisfy this issue?

[stale]

This issue has been automatically marked as stale because it has not had activity for 365 days. It will be closed if no further activity occurs within 14 days. Thank you for your contributions.

[garlick]

I think this issue has been overtaken by other discussions on workflow management, etc.. so closing.