The vault agent sidecar does not reuse from persistent cache the token fetched by the initcontainer agent.

[ozozgun]

Hello,

I have a postgreSQL DB running on VMs that I want to access from a java app running on K8s.
I want to use the dynamic database secrets generated by vault. The dynamic secret configuration works well. I can manually fetch the secrets from vault UI and connect to the DB.
I am now having issue to implement this for my app.
⚠️ I cannot use the Agent Sidecar Injector from https://github.com/hashicorp/vault-k8s, my K8s private cluster does not allow any custom operator.
I am using an initcontainer vault agent with templating for the output of secrets, a persistent cache to share the leases with the sidecar which will later renew the secrets when expired by restarting the app (template command).

Describe the bug

1. I have a initcontainer auto-auth with K8s service token to vault
2. initcontainer fetches the sercrets and putting in a file.
3. The app reads the secrets and starts well.
4. But the sidecar reauthenticate to vault, fetch again the secrets and restarts the application right away.

To Reproduce
here is my agent-config.hcl:

pid_file = "/home/vault/.pid"

auto_auth {
  method "kubernetes" {
    mount_path = "auth/kubernetes"
    namespace = "myVault/OZ_VAULT"
    config = {
      role = "kube-role"
    }
  }
  
  sink "file" {
    config = {
      path = "/home/vault/.vault-token"
    }
  }
}

cache {
  use_auth_auth_token = "true"
  enforce_consistency = "always"
  persist "kubernetes" {
    path = "/vault/agent-cache"
  }
}

vault {
  address = "https://vault-test-local"
  ca_cert = "/etc/pki/unipass-server.pem"
}

template {
  destination = "/vault/secrets/application-vault.yaml"
  contents = <<EOT
    {{- with secret "database/creds/read_write_role" }}
MY_DB_USER: {{ .Data.username }}
MY_DB_PASSWORD: {{ .Data.password }}
  {{ end }}
  EOT
  command     = "/bin/sh -c \"kill -TERM $(pidof java) || true\""
}

I am using the vault image version 1.15.4 both for my vault server and for the agent containers.
Running the containers (init and sidecar) with following command:
vault agent -config=/vault/config/vault-agent-init-config.hcl -exit-after-auth
(the sidecar does not have the -exit-after-auth flag)
Kubernetes version: v1.24.6-mirantis-1

Here is the logs from my initcontainer:

2024-07-10T13:19:23.209+0200 [INFO]  agent.sink.file: creating file sink
2024-07-10T13:19:23.209+0200 [INFO]  agent.sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2024-07-10T13:19:23.209+0200 [INFO]  agent.cache: configured persistent storage: path=/vault/agent-cache
2024-07-10T13:19:23.209+0200 [INFO]  agent.exec.server: starting exec server
2024-07-10T13:19:23.209+0200 [INFO]  agent.exec.server: no env templates or exec config, exiting
2024-07-10T13:19:23.209+0200 [INFO]  agent.auth.handler: starting auth handler
2024-07-10T13:19:23.210+0200 [INFO]  agent.sink.server: starting sink server
==> Note: Vault Agent version does not match Vault server version. Vault Agent version: 1.15.4, Vault server version: 1.15.4+ent.hsm
==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

           Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.15.4, built 2023-12-04T17:45:28Z
             Version Sha: 9b61934559ba31150860e618cf18e816cbddc630

2024-07-10T13:19:23.210+0200 [INFO]  agent.auth.handler: authenticating

And here is my sidecar when it starts:

==> Note: Vault Agent version does not match Vault server version. Vault Agent version: 1.15.4, Vault server version: 1.15.4+ent.hsm
==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

           Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.15.4, built 2023-12-04T17:45:28Z
             Version Sha: 9b61934559ba31150860e618cf18e816cbddc630

2024-07-10T13:19:25.672+0200 [INFO]  agent.sink.file: creating file sink
2024-07-10T13:19:25.672+0200 [INFO]  agent.sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2024-07-10T13:19:25.673+0200 [INFO]  agent.cache: loaded memcache from persistent storage
2024-07-10T13:19:25.673+0200 [INFO]  agent.exec.server: starting exec server
2024-07-10T13:19:25.673+0200 [INFO]  agent.exec.server: no env templates or exec config, exiting
2024-07-10T13:19:25.673+0200 [INFO]  agent.sink.server: starting sink server
2024-07-10T13:19:25.673+0200 [INFO]  agent.template.server: starting template server
2024-07-10T13:19:25.673+0200 [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2024-07-10T13:19:25.673+0200 [INFO]  agent.auth.handler: starting auth handler
2024-07-10T13:19:25.674+0200 [INFO]  agent.auth.handler: authenticating
2024-07-10T13:19:25.674+0200 [INFO]  agent: (runner) creating watcher
2024-07-10T13:19:25.892+0200 [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2024-07-10T13:19:25.892+0200 [INFO]  agent.auth.handler: starting renewal process
2024-07-10T13:19:25.892+0200 [INFO]  agent.template.server: template server received new token
2024-07-10T13:19:25.892+0200 [INFO]  agent: (runner) stopping
2024-07-10T13:19:25.892+0200 [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2024-07-10T13:19:25.892+0200 [INFO]  agent.sink.file: token written: path=/home/vault/.vault-token
2024-07-10T13:19:25.892+0200 [INFO]  agent: (runner) creating watcher
2024-07-10T13:19:25.892+0200 [INFO]  agent: (runner) starting
2024-07-10T13:19:25.895+0200 [INFO]  agent.apiproxy: received request: method=GET path=/v1/sys/internal/ui/mounts/database/creds/read_write_role
2024-07-10T13:19:25.895+0200 [INFO]  agent.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/database/creds/read_write_role
2024-07-10T13:19:25.906+0200 [INFO]  agent.auth.handler: renewed auth token
2024-07-10T13:19:25.913+0200 [INFO]  agent.apiproxy: received request: method=GET path=/v1/database/creds/read_write_role
2024-07-10T13:19:25.913+0200 [INFO]  agent.apiproxy: forwarding request to Vault: method=GET path=/v1/database/creds/read_write_role
2024-07-10T13:19:26.072+0200 [INFO]  agent.apiproxy: received request: method=PUT path=/v1/sys/leases/renew
2024-07-10T13:19:26.072+0200 [INFO]  agent.apiproxy: forwarding request to Vault: method=PUT path=/v1/sys/leases/renew
2024-07-10T13:19:26.073+0200 [INFO]  agent: (runner) rendered "(dynamic)" => "/vault/secrets/application-vault.yaml"
2024-07-10T13:19:26.073+0200 [INFO]  agent: (runner) executing command "[\"/bin/sh -c \\\"kill -TERM $(pidof java) || true\\\"\"]" from "(dynamic)" => "/vault/secrets/application-vault.yaml"
2024-07-10T13:19:26.073+0200 [INFO]  agent: (child) spawning: sh -c /bin/sh -c "kill -TERM $(pidof java) || true"

Expected behavior
I would the sidecar to reuse the leases from the cache created by the initcontainer and restarts the application only the currently used credential expires, instead of fetching new secrets and restarting the app right away.

Additional context
Add any other context about the problem here.

[ozozgun]

the issue was simply that I had a typo in the conf. I fixed it by correcting "use_auto_auth_token=true" instead of "auth_auth"..

I think we should at last put a warning message when we have unknown properties in the agent config file to prevent loosing time for stupid stuff like that ...