implement per-job private KVS namespace

[garlick]

Now that we have @chu11's excellent work on private KVS namespaces merged, it seems useful to implement per-job "guest" namespaces as envisioned in RFC 16. This PR arranges for the job module to create a guest namespace named job<jobid> during job creation, and to "reap" it when the job reaches a terminal state.

During execution, the job may access the the namespace via the FLUX_JOB_KVSPATH environment variable, which is set to the direct namespace path, e.g. ns:job<jobid>/.. The enclosing instance can access it via the lwj.X.Y.<jobid>.guest symlink. After the job completes or fails, the namespace is "reaped", e.g. the symlink is replaced with a snapshot of the final content of the private namespace and the namespace is deleted.

Migrating job data to a private namespace can potentially reduce the bottleneck on serialized commits in the primary KVS namespace and protects the instance against an unintentional KVS DoS by programs that use the KVS heavily.

In this PR, the only data placed in the private namespace is the flux.local_uri and flux.remote_uri values recently added in support of flux wreck uri. It is empty if the job is not a flux instance. In a future PR we may want to migrate stdio and other per-job info.

[grondo]

This is pretty nifty!

I know this is an experiment meant to get us to RFC 16 guest namespace, but since the wreck prototype isn't multi-user capable anyway, I wonder if it would be interesting to have an option to replace the whole job kvs directory with a namespace. This would be interesting because

1. Most of the data in the lwj.x.y.z dirs would normally be under the guest namespace in the exec replacement anyway, so we'd get more representative usage
2. It would allow us to run soak or other tests to see an upper bound in improvement by using namespaces for job kvs data without much work in the other wreck tools we're throwing away anyway
3. It would abstract the kvs path to jobs from the data, e.g. tools could iterate namespaces to get data for jobs instead of iterating kvs dirs.

Maybe the third item above wouldn't really be worth pursuing at this point, but I do find it interesting that the list of current job* namespaces is de-facto a list of currently "active" jobs (with potential for a few stale complete jobs). It would be interesting to leverage that somehow.

Other that that, my only other comment is that I'm confused by the FLUX_JOB_KVSPATH environment variable, which now holds the path to the guest namespace only. Will this be confusing to users that stumble across it and try to use it to get to the actual kvs path? Should it be named FLUX_JOB_KVS_NAMESPACE for clarity, or did I miss the point (very likely)?

[chu11]

Looks pretty good to me. One thought:

+test_expect_success 'schema: guest namespace converted on completion' '
+	flux wreckrun /bin/true &&
+	test_must_fail flux kvs readlink $(last_job_path).guest
+'

should there be the opposite test, to make sure $(last_job_path).guest is a link when the job is running?

[grondo]

should there be the opposite test, to make sure $(last_job_path).guest is a link when the job is running?

Good idea. The wreckrun option --wait-until=<state> is useful for this kind of test (in case the option had slipped by unnoticed)

[garlick]

Thanks for the thoughts guys! I'm especially intrigued by the idea of moving the whole job directory into the namespace. It gets us further along and will probably be a simpler change, even if it doesn't wind up looking quite like RFC 16.

[garlick]

Closing pending rework.