kvs_watch is inefficient and doesn't scale well

[garlick]

The current kvs.watch implementation is inefficient. On receipt of every kvs.setroot message, the kvs module must do an internal "get" on all watcheds keys, compare the old value to the new, then if it has changed, issue a response. It was noted in #802 our current stdio scheme requires flux-wreckrun to register two watchers per task, which creates a lot of work for the rank 0 KVS when a large job is launched.

A possible enhancement is to add a list of changed keys to the kvs.setroot event. The main purpose of the setroot event is to announce a new root reference after a commit. Since the commit logic starts with a list of keys, it would be fairly simple to include that same list in this event. Each kvs module instance would then compare its list of watched keys against the list that changed (and their parent directories which implicitly have changed), and only perform gets on the keys that were included in the commit.

[garlick]

Alternatively, the kvs.watch code could be eliminated from the KVS module, and the client watch implementation could subscribe to the kvs.setroot event internally. To be transactional, there would need to be a way to look up a key against the specific root blobref from the kvs.setroot event, which has been discussed before, e.g. #64.

[chu11]

closing this, this was dealt with via the kvs-watch module, initially merged in #1643