Cannot renew a MongoDB lease due to lease expired

[tonny1983]

How To Reproduce

• Vault Server Configuration

1. Start vault server in dev mode
   vault server -dev

2. Enable app-role
   vault auth enable approle

3. Import a policy
   vault write sys/policy/demo-policy [email protected] (the file is listed later)

4. Create an app role with unlimited tokens and secret ids
   vault write auth/approle/role/readwrite secret_id_ttl=100m token_num_uses=0 token_ttl=100m token_max_ttl=100m secret_id_num_uses=0 policies="default,demo-policy"

5. Enable database secrets vault secrets enable database

6. Configure MongoDB connection info.
   vault write database/config/vault-mongodb-demo-database plugin_name=mongodb-database-plugin allowed_roles="readwrite" connection_url="mongodb://{{username}}:{{password}}@[myip]/admin?ssl=false" username="[myusername]" password="[mypassword]"

7. Configure a role that maps a name
   vault write database/roles/readwrite db_name=vault-mongodb-demo-database creation_statements='{ "db": "vaultdemo", "roles": [{ "role": "dbOwner" }, {"role": "readWrite", "db": "vaultdemo"}] }' default_ttl="1m" max_ttl="1m"

8. Get roleid which will be the value of spring.cloud.vault.app-role.role-id
vault read auth/approle/role/readwrite/role-id

9. Get secret id which will be the value of spring.cloud.vault.app-role.secret-id
vault write -f auth/approle/role/readwrite/secret-id

• demo-policy.hcl file

path "secret/bootstrap" {
	capabilities = ["create", "read", "update", "delete", "list"]
}
path "secret/application" {
	capabilities = ["create", "read", "update", "delete", "list"]
}
path "database/creds/readwrite" {
	capabilities = ["create", "read", "update", "delete", "list"]
}
path "database/creds/readwrite/*" {
        capabilities = ["create", "read", "update", "delete", "list"]
}
path "sys/leases/*" {
	capabilities = ["create", "read", "update", "delete", "list"]
}

• bootstrap.yml file

spring:
  cloud:
    vault:
      host: localhost
      port: 8200
      scheme: http
      uri: http://localhost:8200
      connection-timeout: 5000
      read-timeout: 15000
      config:
        lifecycle:
          enabled: true
        order: -10
      authentication: APPROLE
      app-role:
        role-id: [from step 8]
        secret-id: [from step 9]
      mongodb:
        enabled: true
        role: readwrite
        backend: database
        username-property: mongodb.username
        password-property: mongodb.password

• Components version

1. Spring Boot 2.0.6.RELEASE

2. Spring Cloud Finchley.SR1

3. Spring Vault core 2.1.1.BUILD-SNAPSHOT

4. Vault v0.11.4

Error Description

After about 1 minute from starting the demo app, I got the following exception:

2018-10-25 22:11:26.616  WARN 5446 --- [g-Cloud-Vault-2] LeaseEventPublisher$LoggingErrorListener : [RequestedSecret [path='database/creds/readwrite', mode=RENEW]] Lease [leaseId='database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V', leaseDuration=PT10S, renewable=true] Cannot renew lease: lease expired

org.springframework.vault.VaultException: Cannot renew lease: lease expired
	at org.springframework.vault.core.lease.SecretLeaseContainer.doRenewLease(SecretLeaseContainer.java:604) [spring-vault-core-2.1.1.BUILD-20181010.071554-4.jar:2.1.1.BUILD-SNAPSHOT]
	at org.springframework.vault.core.lease.SecretLeaseContainer.lambda$scheduleLeaseRenewal$0(SecretLeaseContainer.java:503) [spring-vault-core-2.1.1.BUILD-20181010.071554-4.jar:2.1.1.BUILD-SNAPSHOT]
	at org.springframework.vault.core.lease.SecretLeaseContainer$LeaseRenewalScheduler$1.run(SecretLeaseContainer.java:776) ~[spring-vault-core-2.1.1.BUILD-20181010.071554-4.jar:2.1.1.BUILD-SNAPSHOT]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.0.10.RELEASE.jar:5.0.10.RELEASE]
	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93) ~[spring-context-5.0.10.RELEASE.jar:5.0.10.RELEASE]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_152]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_152]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_152]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_152]

And error message also shows from vault server:

2018-10-25T22:11:26.607+0800 [ERROR] secrets.system.system_79b963f3: lease renewal failed: lease_id=database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V error="lease expired"
2018-10-25T22:11:26.690+0800 [INFO]  expiration: revoked lease: lease_id=database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V

Besides, before the exception, there's some info of spring vault scheduling the renewal:

2018-10-25 22:10:26.506 DEBUG 5446 --- [           main] o.s.v.c.e.LeaseAwareVaultPropertySource  : Requesting secrets from Vault at database/creds/readwrite using RENEW
2018-10-25 22:10:26.558 DEBUG 5446 --- [           main] o.s.v.core.lease.SecretLeaseContainer    : Secret database/creds/readwrite with Lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V qualified for renewal
2018-10-25 22:10:26.559 DEBUG 5446 --- [           main] cretLeaseContainer$LeaseRenewalScheduler : Scheduling renewal for secret database/creds/readwrite with lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V, lease duration 60
2018-10-25 22:10:36.561 DEBUG 5446 --- [g-Cloud-Vault-1] cretLeaseContainer$LeaseRenewalScheduler : Renewing lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6Vfor secret database/creds/readwrite
2018-10-25 22:10:36.570 DEBUG 5446 --- [g-Cloud-Vault-1] o.s.v.core.lease.SecretLeaseContainer    : Secret database/creds/readwrite with Lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V qualified for renewal
2018-10-25 22:10:36.570 DEBUG 5446 --- [g-Cloud-Vault-1] cretLeaseContainer$LeaseRenewalScheduler : Scheduling renewal for secret database/creds/readwrite with lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V, lease duration 50
2018-10-25 22:10:46.574 DEBUG 5446 --- [g-Cloud-Vault-2] cretLeaseContainer$LeaseRenewalScheduler : Renewing lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6Vfor secret database/creds/readwrite
2018-10-25 22:10:46.578 DEBUG 5446 --- [g-Cloud-Vault-2] o.s.v.core.lease.SecretLeaseContainer    : Secret database/creds/readwrite with Lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V qualified for renewal
2018-10-25 22:10:46.579 DEBUG 5446 --- [g-Cloud-Vault-2] cretLeaseContainer$LeaseRenewalScheduler : Scheduling renewal for secret database/creds/readwrite with lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V, lease duration 40
2018-10-25 22:10:56.579 DEBUG 5446 --- [g-Cloud-Vault-1] cretLeaseContainer$LeaseRenewalScheduler : Renewing lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6Vfor secret database/creds/readwrite
2018-10-25 22:10:56.585 DEBUG 5446 --- [g-Cloud-Vault-1] o.s.v.core.lease.SecretLeaseContainer    : Secret database/creds/readwrite with Lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V qualified for renewal
2018-10-25 22:10:56.585 DEBUG 5446 --- [g-Cloud-Vault-1] cretLeaseContainer$LeaseRenewalScheduler : Scheduling renewal for secret database/creds/readwrite with lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V, lease duration 30
2018-10-25 22:11:06.587 DEBUG 5446 --- [g-Cloud-Vault-2] cretLeaseContainer$LeaseRenewalScheduler : Renewing lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6Vfor secret database/creds/readwrite
2018-10-25 22:11:06.592 DEBUG 5446 --- [g-Cloud-Vault-2] o.s.v.core.lease.SecretLeaseContainer    : Secret database/creds/readwrite with Lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V qualified for renewal
2018-10-25 22:11:06.592 DEBUG 5446 --- [g-Cloud-Vault-2] cretLeaseContainer$LeaseRenewalScheduler : Scheduling renewal for secret database/creds/readwrite with lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V, lease duration 20
2018-10-25 22:11:16.596 DEBUG 5446 --- [g-Cloud-Vault-1] cretLeaseContainer$LeaseRenewalScheduler : Renewing lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6Vfor secret database/creds/readwrite
2018-10-25 22:11:16.600 DEBUG 5446 --- [g-Cloud-Vault-1] o.s.v.core.lease.SecretLeaseContainer    : Secret database/creds/readwrite with Lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V qualified for renewal
2018-10-25 22:11:16.600 DEBUG 5446 --- [g-Cloud-Vault-1] cretLeaseContainer$LeaseRenewalScheduler : Scheduling renewal for secret database/creds/readwrite with lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V, lease duration 10
2018-10-25 22:11:26.603 DEBUG 5446 --- [g-Cloud-Vault-2] cretLeaseContainer$LeaseRenewalScheduler : Renewing lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6Vfor secret database/creds/readwrite
201

I'm using the version of 2.1.1.BUILD-SNAPSHOT because before the version I got 403 forbidden error when renewing the secret just as the question and the issue #255 described, and using the version according to the comment in #255 . However, it seems the problem reported in spring vault #319 still occurs in spring cloud vault.

[mp911de]

Thanks a lot for the detailed report. Credentials requested by Spring Cloud Vault are not rotated. They are only renewed. The reason is that properties are exposed as config properties and the components that use configuration properties do not expect a change of properties during runtime.

In the example above, we see log output that keeps the secret alive (renewal). Once the terminal TTL expires, the last renewal fails correctly with an error.

You can request a rotating secret for MongoDB credentials but then you are in charge to control the MongoDB client lifecycle. There is no possibility to check whether MongoDB or other, similar resources, are currently executing queries so the underlying resource could be swapped with a fresh one.

See also:

• Renewing AWS Backend credentials (or getting new ones) #85
• https://stackoverflow.com/questions/52950107/automatically-renew-aws-credentials-in-a-spring-boot-application-using-spring-cl/52987419#52987419

[tonny1983]

Thanks a lot for your reply. But I'm afraid the case is nothing with secret rotation. According to the vault guide Create Role, the default_ttl and max_ttl properties are all TTL for the leases but secret. The Spring Vault/Spring Cloud Vault should renew the lease in its TTL which is ONE minute in the case.

[mp911de]

Renewal is that what you're seeing in the logs. Once a lease exceeds max_ttl, Vault revokes the secret and it cannot be renewed anymore. This is the reason, why we see Status 400 when the renewal tries to renew the lease after about a minute.

[tonny1983]

Thus, in this case, I should rotate a new lease to replace the revoked timeout lease, and I should be in charge to control the new lease. Am I right?

[mp911de]

Yes. See the CustomConfiguration sample for more details. You also need to take care of your MongoClient reconfiguration.

May I close this ticket or is there anything else I can assist you with?

[tonny1983]

I got it. And may I have just one more further question please? How about the secret id when it reaches the TTL? Should I also take charge to control the rotation of secret id?

[mp911de]

I assume you're talking about secretId in the context of AppRole authentication. So, this depends primarily on your configuration. In general, Spring Vault uses authentication mechanisms to obtain a session token that can be TTL'd. If the session token expires, Spring Vault attempts to re-login using the provided credentials. If authentication mechanism components are TTL'd, as well (JSON Token, identifiers, …) then Spring Vault can only re-login until the authentication component's TTL expires.

If you provide a static secret Id with a TTL, then Spring Vault can re-login within the TTL timeout. After that time, you need to provide a new secret Id. If you use pull-mode for the secret Id, then Vault can obtain a fresh secret Id itself and properly re-login.

Closing this one as the questions are answered.

[usr42]

@tonny1983 I know that the thread is a bit old already. I hope it it is still helpful for you or other people stumbling across the same question.
I've written a blog post about how to ensure that expiring Spring Cloud Vault dynamic database secrets are renewed, when reaching Hashicorp Vault's max_ttl: Hashicorp Vault max_ttl Killed My Spring App
There will be a follow-up post which is more technology specific (HikariCP) but renews the PostrgreSQL credentials on the fly, which also works for really short-lived secrets (I've tested e.g. with max_ttl of 1 minute).
If there is the demand I could also check how something similar could be achieved for MongoDB and then I would write another blog post with that information.

[usr42]

@tonny1983 The follow-up post about how to rotate expiring relational Spring Cloud Vault database credentials without downtime is available: Heavy Rotation of Relational Hashicorp Vault Database Secrets in Spring

[scyellleader]

@usr42 I would like to see a blog post on rotating MongoDB secret.

[sullrich84]

@mp911de, @usr42 apparently this is still the case in 2022.

Unfortunately, it is difficult to see from the documentation that dynamic secrets, which are probably the most common reason for using vault, cannot be automatically rotated from spring-cloud-starter-vault-config and spring-cloud-vault-config-databases.

However, the following passage can be found in the current documentation:

enabled controls whether leases associated with secrets are considered to be renewed and expired secrets are rotated. Enabled by default.

Do you think it would be a useful extension to mention these restrictions in the documentation?

[mp911de]

Good point, it makes sense to reword the sections to reflect that rotation works only in some cases (such as AWS).