need a way to run across all resources allocated to a job

[trws]

This is related to #866, but some way to run across an entire inner instance would be really good in support of job steps. I think @grondo mentioned something to this effect at the last meeting, but I just ran up against not having this and had to build a workaround for splash, it's something we'll have to have at some point.

[grondo]

Yeah this is in the use cases we documented -- see Use Case 4.4 here.

I keep meaning to submit a PR for that particular Use Case to document your idea to make that behavior (select all resources) an option instead the default behavior within a job. This solves many problems nicely (flux-run or whatever utility doesn't need a heuristic to decide when to run across all resources, and the default behavior is the same inside or outside of a "batch script", which makes it more consistent)

[trws]

As a note, for now all such jobs have to run in an inner instance and I'm using this to generate a hostlist:

flux kvs get -j $(flux kvs dir resource.hwloc.by_rank | sed -e 's/$/.HostName/')

[trws]

I should say, "have to run in an inner instance for now" until we have a way to avoid it.

[grondo]

Just so I understand, something like flux wreckrun -N $(flux getattr size) doesn't also work to run across the entire instance?

Are you using the generated hostlist for input to mpirun or mpiexec or similar?

The hostname list could also be generated with flux exec -r all hostname, however the flux kvs usage above is probably more efficient. In an rc script you could probably pre-fetch this and store it to a kvs key. If you wanted it in "hostlist" format, filter the output through hostlist --sort.

[trws]

I am at the moment, it's because of the spectrum MPI issue #1382. Wreckrun like that works for launching non-MPI things on there right now, but I didn't think of the getattr option for size, that's a much better way to do it. Maybe putting it under an option on wreckrun/submit would be a quick way to deal with this.

[grondo]

closed by #1988