config: need persistent storage option(s)

[garlick]

Problem: the default "rundir" for the system instance is /run. The systemd unit file manages a subdirectory of /run for flux that goes away when flux is stopped, which removes anything written out during rc3 teardown.

The system instance will likely need to save and restore the content cache, last KVS root blobref, logs, and (future?) job info database.

The only way we currently have of changing this behavior is setting the following broker attributes:

persist-directory
A persistent directory available for storage on rank 0 only.
If persist-directory is not defined, persistence is unavailable
and users should fall back to broker.rundir, with cleanup.

persist-filesystem
If defined, and persist-directory is not defined, the rank
0 broker chooses a unique name for persist-directory within
persist-filesystem and creates it automatically.

We need reasonable TOML config file support for redirecting storage to a persistent area, for example in /var. The sematics of these attributes are a bit confusing, so IMHO the whole scheme needs a rethink.

[garlick]

Let's call this closed by #2783.