agent: persistent caching support

[tvoran]

Adds the option of a write-through cache to the agent, backed by boltdb, with the aim of preserving the renewal of leases and tokens across agent restarts.

This functionality is configured through a new persist stanza within the cache portion of the vault agent config file:

cache {
  persist "kubernetes" {
    path = "/path/to/db/"
    exit_on_err = false
    keep_after_import = false
    service_account_token_file = "/path/to/jwt"
}

• The persist type specifies the key protection scheme to use for the key encrypting/decrypting the persisted data. Currently "kubernetes" is supported, where the encryption key is saved in the boltdb, using the service account jwt as the AAD (extra data) used to encrypt and decrypt.

  • service_account_token_file is optional, and defaults to "/var/run/secrets/kubernetes.io/serviceaccount/token"

• The path parameter is required, and is the file system location to write the cache db file. The file name for the cache file is vault-agent-cache.db. So in the above example the full file and path would be /path/to/db/vault-agent-cache.db.

  • If the db file exists at path when the agent starts, anything in the db file will be imported into the vault-agent in-memory cache. Otherwise a new db file is created.

• If keep_after_import = false, the db file will then be deleted after importing, and agent tokens and secrets will only be cached in memory.

• If exit_on_err = true, errors during import of an existing db cache will cause the agent to exit.

The bolt db is laid out in multiple buckets:

• The "meta" bucket contains a version: 1 entry for use in the future if migrations are required. It also contains the most recent auto-auth token cached by the agent, and the key material used to decrypt and restore an existing cache. This key material may either be the key, or some other token used to retrieve the key.
• There are three buckets for the different cached item types: "token", "auth-lease", and "secret-lease". They're separated this way due to dependencies during the restore process (e.g. secret-leases depend on token's and auto-lease's).

[calvn]

Looks like this bucket is carved out for the auto-auth token, but can that be stored in the root directly?

[tvoran]

Perhaps? Though I left it as a separate bucket since this is how the cachememdb behaves, namely old auto-auth tokens are not removed if a new one comes in, and they aren't expired. So I added this bucket out of abundance of caution, in case a lease that needed to be restored relied on a previous auto-auth token.