programmable sync points between jsc and rexec

[dongahn]

Moving from #206 to capture the use cases and discussions on fine-grained programmable sync points support between jsc and rexec:

5. For notify-control pattern, we talked about the idea of fine-grained programmable sync points support within rexec. That is, users of jsc can add some behavior (e.g., stop) on a programmable set of events for a particular job. We at least agreed that using these sync points for all of the jobs will incur unneccessarily high overheads and we want to do this in a fine-grained manner if we do this.

If this is combined with the new eventing scheme proposed in #206, the user will probably have to send a "state control mask" (the set of states by which the user want rexec to be synchronized) as part of a new job creation request. I was further talking to @SteVwonder, and it seems it is kind of premature to decide how sync points ideas can be materialized. When we have more concrete use cases with runtime tools and dynamic scheduling implementation, we may know more about what's needed.

[garlick]

How is this like/unlike Slurm SPANK plugins?

[dongahn]

@garlick Thanks, I will take a look at it.

[dongahn]

Just to further the discussion a bit:

I took a look at the wreck services to understand where and how those states are emitted.

• next-id is incremented by the job module.
• starting is generated by wrexecd right before the wrexecd session fork/exec the target program processes
• after the fork/exec, either sync or running is generated: sync is debugger support -- spawning the target program processes and leave them in a stop state (using ptrace); running is course to simply let loose these processes.

The state set will be richer when we get to the real rexec service, and this seems a good time to think about what/how synchronization support we need to provide to the users of these states (e.g, jsc) a bit more concretely. BTW, this was discussed a bit in #206: In general, I like the idea of making things "asynchronous" to the services and "synchronous" to the job as much as possible.

Some use cases:

• pre-staging tool wanting to stage its program before the target program is spawned (e.g., I/O forwarding layer? -- CEA's IO-forwarding requirement might give us some concrete idea about what is the sync point needs of their tool.
• debugging tools: to control the program execution from the beginning, the sync point can be a bit later than these pre-staging tools. It can wait until the target program fork/exec'ed/stopped and the current sync support should be sufficient. However, if users want to apply those tools on some other events like aborting, we need sync support on that event. Some mechanisms to keep the aborting processes from exiting so that the debugger can be attached. Or Some mechanisms to hook an programmable core file generation. This would be similar to CRAY ATP support. Further, we will need to determine the necessary sync operations for grow/shrink when we start supporting dynamic scheduling. The debugger will need to be sync'ed with a new set of processes grow fork/exec's and be notified of a set of processes that are killed by shrink.
• performance tools: their sync need will be similar to debugging tools. But they might not have ptrace like capability to sync with the job... (a bit vague at this point though.)
• dynamic scheduling: The requirement can only be exactly known when we lay the foundation later this summer.

Using the current wrexec implementation as a simple model for rexec. The above sync points are translated to:

• pre-staging tool: sync right after wrexecd emit starting. You want to make sure wrexecd won't go ahead and fork/exec the program. If this is the good sync point or not really depends on the concrete use cases
• debugging tools: sync is already provided between starting to running with ptrace so the initial case is covered. abort state isn't available yet, so we will need to defer that discussion. How wrexecd will be sync'ed with the target program would be interesting in and of it self. (If MPI, we can have some hook in MPI_Abort to create a sync point with wrexecd.) sync needs for grow and shrink are also something that we need to discuss when dynamic scheduling comes.

[grondo]

closed by #1988