K/V v2: add suport for lease_duration using ttl key

[asaintsever]

Is your feature request related to a problem? Please describe.
Using K/V v2, which is enabled by default with Vault server in dev mode, adding a ttl key has no effect on lease_duration attribute (remains 0). With K/V v1, ttl key allows to set lease duration which is very handy. It seems that with K/V v2 we cannot benefit from this feature.

Describe the solution you'd like
Same special handling of ttl key as we currently have in K/V v1 (see https://www.vaultproject.io/docs/secrets/kv/kv-v1.html#ttls)

Describe alternatives you've considered
N/A

Explain any additional use-cases
Cannot use the versioned version of K/V (i.e. v2) if we want to set a ttl and have consul template refresh its template based on lease_duration value (and not vault token ttl). Stuck on K/V v1 for the moment.

Additional context
N/A

[briankassouf]

duplicate of #6184

[seh]

@briankassouf, I don't think this is a duplicate of #6184. At present, if one runs vault kv patch ttl=60s, you wind up with a value vault.Secret.Data["data"]["ttl"] = "60s", which is placing the TTL value in the secret payload, not in its metadata.

My goal is to populate vault.Secret.Data["ttl"] for KV v2 secrets, so that (*vault.Secret).TokenTTL picks up the value here.

@asaintsever, do you concur?

[asaintsever]

@seh yes totally, for KV v2, a "ttl" secret value should end up in Data["ttl"] (as it is the case with KV v1), not in Data["data"]["ttl"] where it is not considered currently (as you pointed out: https://github.com/hashicorp/vault/blob/master/api/secret.go#L263-L272).

[takaishi]

lease_duration is used in consul-template (https://github.com/hashicorp/consul-template/blob/master/dependency/vault_common.go#L126-L167) to wait next update.
But consul-tempalte's behavior is not compatible for K/V v1 and K/V v2 now, consul-template wait default ttl (5 minutes).

When enable secret and put data as following:

vault secrets enable -version=1 kv-v1
vault secrets enable -version=2 kv-v2
vault kv put kv-v1/with-ttl a=111 ttl=60s
vault kv put kv-v2/with-ttl a=111 ttl=60s

When I get from kv-v1, lease_duration is same value with ttl field.

vault kv get -format=json kv-v1/with-ttl
{
  "request_id": "b30f388d-b12b-f37d-b491-05d7ea04c0a2",
  "lease_id": "",
  "lease_duration": 60,
  "renewable": false,
  "data": {
    "a": "111",
    "ttl": "60s"
  },
  "warnings": null
}

But I get from kv-v2, inspite of ttl field exists, lease_duration is 0.

vault kv get -format=json kv-v2/with-ttl
{
  "request_id": "2a51dd87-0995-c493-59ea-38432e6ed4b3",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "data": {
      "a": "111",
      "ttl": "60s"
    },
    "metadata": {
      "created_time": "2020-03-17T02:52:12.804263Z",
      "deletion_time": "",
      "destroyed": false,
      "version": 1
    }
  },
  "warnings": null
}

lease_duration field is updated from input's Secret field in LogicalResponseToHTTPResponse, but input doesn't has Secret when K/V v2.

https://github.com/hashicorp/vault/blob/master/sdk/logical/translate_response.go#L14-L46

K/V v1 inputs's Secret field inserted in following code:
https://github.com/hashicorp/vault-plugin-secrets-kv/blob/master/passthrough.go#L166-L184

K/V v2 inputs's Secret field does'nt inserted.
https://github.com/hashicorp/vault-plugin-secrets-kv/blob/master/path_data.go#L84-L170

If K/V v1 and K/V v2 should be have lease_duration from ttl field, I ready to send a patch.

[mimperatore]

Should this issue be reopened?

@seh felt that this was not a duplicate of #6184, and I tend to agree, because that issue is about destroying the secret after the TTL, whereas this request is about providing equivalent support as v1 for the lease duration.

@takaishi provided some ideas about how to address this, and was willing to put a PR together.

Without support for this feature, I am reluctant to adopt kv-v2 because it doesn't give me the ability to auto-renew my secrets automatically (for example, when I want to use the vault agent injector sidecar).

[Chrisss93]

This has become a major issue for us as well. Is the lease_duration concept being deprecated? If not what is the advised way of setting lease_duration on KV v2 engine secrets? We are considering moving our engines back to KV v1 if we are unable to set the lease_duration in v2.

[voiprodrigo]

Hello. Can this issue be re-opened, please? It was incorrectly closed by @briankassouf as a duplicate of #6184.
Thanks.

[huntc]

Requesting that this issue be re-opened also. That, or the documentation at https://www.vaultproject.io/docs/secrets/kv/kv-v1#ttls be updated to reflect v2 behaviour. Thanks.

[heatherezell]

Hi folks - we hear you! I'll re-open this and engage the engineers in another discussion about it. Thanks for your patience!

[ccapurso]

Hi, everyone! The Vault team has discussed this internally in order to identify a path forward. The ttl and lease_duration option for KVv1 is a workaround that we do not wish to perpetuate to KVv2. The various options of TTL support that we have discussed includes deletion and cache control.

The concept of deletion has been addressed via the delete_version_after KVv2 metadata field. As mentioned in multiple comments, this does not address the ttl field in KVv1 which ultimately maps to lease_duration in the response. The Vault team's preferred solution would be to properly support cache control via appropriate response headers. Our proposals are still under review, however. That said, whatever option is chosen will need to be used within consul-template as well to provide a full solution when this work is prioritized.

We will keep this issue open and be sure to keep everyone updated with our status. We appreciate all of your feedback!

[eikenb]

The Vault team's preferred solution would be to properly support cache control via appropriate response headers.

How will this be made available through the API? Lease duration is made available on the data structure returned. How will API clients be able to access this value to determine how long to wait when polling secrets?

[huntc]

The Vault team's preferred solution would be to properly support cache control via appropriate response headers.

How would TTL be expressed when creating/updating a secret?

[the-maldridge]

Any update on this limitation? I find myself reminded of this roughly every 70 days when letsencrypt certs roll over, are reloaded into vault, and then not updated by consumers.

[thallesdaniell]

To handle K/V v2 lease I am using static_secret_render_interval and it has been working for me temporarily.

config.hcl: |
    template_config {
      exit_on_retry_failure = true
      static_secret_render_interval = "30s"
    }
    
    template {
      contents = "{{- with secret \"staging/data/database/config\" -}}{{- range $k, $v := .Data.data }}{{ printf \"%s=\\\"%s\\\"\\n\" $k $v }}{{- end }}{{- end }}"
      destination = "/vault/secrets/.env"
    }

[the-maldridge]

That's a good shout. Unfortunately I'm running into this problem inside Nomad where that key isn't exposed. I've opened hashicorp/nomad#17423 to see if I can gain access to that key inside the nomad environment.