kvs vs kvs-watch: handling namespaces consistently

[chu11]

While working on #1838 I realized a subtlety in kvs-watch behavior vs kvs. It's subtle, and in some cases may not matter, but I think we need to make a decision on what we want in what situations.

In the old kvs.lookup and kvs.watch, the following are considered identical:

flux kvs --namespace=foo get a
flux kvs get ns:foo/a

In other words, each mechanism is considered identical to specify a namespace.

But there's differences with kvs-watch.

For example, when monitoring, the namespace specified via --namespace=foo (internally this calls flux_kvs_set_namespace()), this is the namespace that is actively "monitored" and whatever text is in the "key" is the actively watched for. So for example:

flux kvs --namespace=foo get --watch mykey
flux kvs put ns:foo/mykey=1

does not work in the way some might think it should work (Edit: I believe it's b/c setroot sends the key "ns:foo/mykey", which ends up not strcmp matching the original key "mykey").

I thought:

flux kvs get --watch ns:foo/mykey
flux kvs put ns:foo/mykey=1

would work, but flux kvs get --watch ns:foo/mykey actually returns EINVAL, which I don't know why at the moment.

We did discuss some namespace API kinds of things in #1435 and perhaps we need to resolve those first.

[chu11]

Throwing out one possible idea. We could consider --namespace (or the environment variable FLUX_KVS_NAMESPACE) as the only way(s) to specify a namespace. When a user specifies a key w/ a "ns:" prefix, that is always treated as a key with namespace hopping embedded.

So in my example above:

flux kvs get --watch ns:foo/mykey
flux kvs put ns:foo/mykey=1

The user is actually watching for the key "ns:foo/mykey" on the primary namespace. They are not watching on the "foo" namespace. Thus:

flux kvs get --watch ns:foo/mykey
flux kvs --namespace=foo put mykey=1

would not work (discounting possibly when the user uses the --full option).

This is just an idea, and I'm a little unsure how this would play out in code.

[chu11]

flux kvs get --watch ns:foo/mykey

Returns EINVAL b/c internally kvs-watch passes a root reference on the primary namespace at the same time specifying the foo namespace as the desired namespace. kvs considers this an EINVAL combination.

[chu11]

Another consideration is that we handle #1447 first too. Lets say

flux kvs get --watch ns:foo/dir.a

is allowed. What if the user does

flux kvs put ns:foo/////dir......a

?

[chu11]

possible solution:

kvs-watch needs to process "ns:" prefix and normalize keys for consistency to the main kvs module. That solves consistency issue 1.

whenever keys are processed/sent around they should never have the "ns:" prefix, and they should always be normalized. I think that solves much of the inconsistency of kvs-watch matching keys. So if the user inputs "ns:/foo/bar", kvs-watch should only be looking for the key "bar". And when a setroot is sent, it should never send "ns:/foo/bar", it should send "bar".

I believe this requires kvs transactions to strip "ns:" prefix before storing keys in a transaction. This is acceptable, b/c crossing namespaces is illegal in a transaction.

The big negative about this is (may be?) that it would require all keys to be pre-processed and checked for namespace crossing at the beginning of a transaction being processed. This is the opposite of what goes on now, where if an illegal namespace cross occurs in a kvs transaction, it is detected during processing.

[garlick]

Is it possible to detect namespace crossings in advance of beginning a transaction? What if a symlink with an ns: prefix is encountered during the namespace walk?

[chu11]

@garlick ahh, good point. Here I think we only care about namespace crossing via the keys listed in the transaction, b/c that is what we send in the setroot event. Edit and I believe that can be pre-processed ahead of time.

Like if the keys are: "A, B, ns:foo/bar", we only care if the namespace "foo" is different than the namespace we are in. If we happen to be in the namespace foo, then we alter this to be "A, B, bar" for the setroot event. If we aren't in the namespace foo, we generate an error.

If one of those keys happens to be a symlink and that crosses namespaces, an error occurs in the transaction. And if one of those keys happens to be a symlink, but it doesn't cross a namespace, then the key in the setroot event doesn't change.

[garlick]

This seems reasonable - in fact, could we limit the use of ns: keys to lookups, and just issue an errror if they occur in a commit? (except as the target of a symlink?)

[chu11]

This seems reasonable - in fact, could we limit the use of ns: keys to lookups, and just issue an errror if they occur in a commit? (except as the target of a symlink?)

I suppose we could. It would make a lot of code simpler. But that feels inconsistent to not support it in writes.

Here's a thought, allow the ns prefix only in symlinks? That is really the predominant reason we supported it? It's sort of just convenience everywhere else.

[garlick]

It feels like if we support the namespace prefix in symlnks, we are creating the expectation that it should work in lookup. Maybe it would be better to have a dedicated interface for creating that type of link, like flux_kvs_txn_namespace_symlink(txn, flags, key, namespace, target) and maybe a new tree object for it and do away with the key prefix entirely?

[chu11]

Maybe it would be better to have a dedicated interface for creating that type of link, like flux_kvs_txn_namespace_symlink(txn, flags, key, namespace, target) and maybe a new tree object for it and do away with the key prefix entirely?

I think that would be nice. A lot of the coding to handle the special "ns:" prefix is all over the place. Having it be a special treeobj would simplify a lot of things. Perhaps should be a new issue once #1859 is looked at.