flux-kvs: improve flux kvs dir performance

[chu11]

flux kvs dir retrieves all keys and values for a directory. It truncates large values, which means that very large values (think stdout from a job) can make flux kvs dir slow, even if the output value won't be displayed.

non-thought out brainstorming ideas

• the new FLUX_KVS_STREAM would improve performance of large values, although it would still suffer large values being transmitted
• some mechanism to tell FLUX_KVS_STREAM stop after X bytes? Or after 1 blob?
• ???

[chu11]

some mechanism to tell FLUX_KVS_STREAM stop after X bytes? Or after 1 blob?

It occurs to me that the "bytes" part of this is impossible, since kvs-watch wouldn't know how much data there is until the content blob is retrieved. Perhaps something related to the KVS data size stuff (#6264 #6265) could maybe solve this in the future.