flux kvs dir command should not list values by default

[garlick]

flux kvs dir displays values as well as keys by default:

Usage: flux-kvs dir [-R] [-d] [key]
Display all keys under directory
  -R, --recursive        Recursively display keys under subdirectories
  -d, --directory        List directory entries and not values
  -h, --help             Display this message.

This can result in fairly unreadable output when the KVS contains complex values, for example the XML blob stored as a JSON string by resource-hwloc or base64 stdio data stored as a sequenced directory of JSON strings by wrexecd via libkz.

Furthermore, after #1154 converts the KVS metadata to RFC 11 form, the assumption that all values are JSON no longer holds, so displaying a value without hints from the user will be a dodgy proposition.

Finally, I'm not aware of an actual use case for dumping values via flux kvs dir.

My vote would be to make flux kvs dir behave more like ls(1). In fact, if we want to subliminally suggest to users that flux kvs dir options behave they way an ls(1) user would expect, it might not be a bad idea to rename it to flux kvs ls.

[chu11]

FWIW, I think displaying the keys & values is useful for "discovery". There's so much in the KVS, a lot of times you sort of want to just see what's there. I've certainly used it when debugging just to see what I may have screwed up along the way.

I don't know if this is enough to warrant a --output-json kind of option. But I can see wrapper scripts to do the equivalent being written up.

[grondo]

Would it be too much to keep a flux-kvs dir and add flux-kvs ls which works like ls(1) as @garlick suggests? (I was going to suggest something similar a long time ago, and I think an ls like command is much more intuitive than the dir command we have now -- though that command is useful for debug as @chu11 said)

[garlick]

I'm fine with retaining that capability if it's useful. Maybe add an ls subcommand and rename dir to dump?

As mentioned above it's trickier to decode the values into something printable after #1154 though. Maybe the default could be to directly display the raw value (which might be encoded JSON), with non-printable characters converted to "." and output truncated so that key: value fits in 80 columns?

[chu11]

I like the idea of calling it dump if a ls is implemented.

[garlick]

I was just thinking, maybe a flux kvs tar command that can dump/restore a snapshot of a particular directory/namespace would be more useful here? E.g.

flux kvs tar --create --file ARCHIVE key [key ...]
flux kvs tar --extract --file ARCHIVE

For exploration purposes, you could either do

flux kvs tar --create --file - dir | tar xvf -

and explore the file hierarchy that is created, or

flux kvs tar --create --file - dir | tar tvf -

and review the table of contents in a familar form?

Seems like such a thing would be more generally useful than the proposed dir/dump.

The POSIX tar archive format is simple and well documented:

https://www.gnu.org/software/tar/manual/html_node/Standard.html

[grondo]

The tar idea is interesting, but it doesn't solve the use case for dir/dump, which is to show all the keys and their values for a directory for quick debugging purposes.

Also, wouldn't it be somewhat more straightforward to allow a flux kvs restore that takes a kvs-specific dumpfile format from flux kvs dump than to try to kludge the tar archive format to work? (i.e. there are a lot of fields that don't make sense like uid, gid, mtime, mode, etc...). With dump you could emit a dumpfile version and add features in the future...