Support using tokens with vault agent for local development

[chilversc]

I want to use vault agent to automatically refresh my token and render templates when developing locally to avoid storing passwords for services such as CloudSQL (even for development environments).

Currently when developing I have to manually renew the token and other credentials or my various applications / docker containers come to an abrupt halt when the leases expire.

It seems vault agent is supposed to be the service that deals with this issues by managing the tokens for you, and refreshing tokens / secrets as required. However, I can't use the vault agent with the standard auto-auth methods as I'm using oidc to authenticate with vault.

The process I wanted to set up would be:

• Log in to vault using oidc (or some other method) to obtain a token
• Start vault agent (locally / docker container / minikube)
• Vault agent picks up the vault token and starts managing it (auto-renew)
• Vault agent renders any templates and starts managing them
• Start applications (locally / docker container / minikube)

At some point the token will reach its maximum lease time and cannot be renewed. The maximum lease time is expected to be sufficient for general development. Once the token expires the user will need to re-authenticate and restart the agent.

As an alternative to restarting the agent, it could support monitoring a file (similar to how jwt auto-auth works). Then after re-authenticating the user can write the token out to the file that the agent is monitoring.

[chilversc]

So I've subsequently discovered that consul-template can handle this on its own once you've signed in to vault using oidc. Though this wasn't obvious as you do not come across the consul-template documentation until you get to the documentation on setting up vault with kubernetes.

When you first search for setting up this, most of the links talk about vault agent. So perhaps this is mostly a documentation issue?

It would be nice if https://www.vaultproject.io/docs/agent/ would mention consul template and that consul template can be used standalone (and will refresh the tokens). The current docs makes it sound like vault agent is responsible for refreshing the tokens, and just uses consul-template as a library to generate the output.

The main issues I found was that:

I'm not yet using consul so wouldn't have read that documentation, and its easy to skip as it seems unrelated to vault when your getting started. The initial documentation suggests that consul is more for storage, etc, later when deploying vault for production.

If you do not read the consul documentation, its not obvious that consul template is a stand-alone application that has the capability to manage and refresh vault tokens. From the name it sounds like its part of consul, and that its just a simple template engine (file in, file out).

Keeping it simple, and focusing on one part at a time (e.g. running vault locally, not worrying about kubernetes until later) means you do not read the documentation that actually mentions consul-template can handle this. Most searches on the internet will end up referring you to vault agent, and when looking through the vault docs (instead vault learn) its the most obvious section that stands out.

[krzysztof-miemiec]

@chilversc have you managed to somehow set up Vault Agent locally with OIDC?

Edit: I read your post one more time and found the solution 😅

1. Download https://github.com/hashicorp/consul-template
2. Write your secret templates as usual
3. Define hcl config file for consul-template:
   templates.hcl:

pid_file = "./configs_pid"

vault {
  address = "https://vault.example.com"
}

template {
  source               = "common.ctmpl"
  destination          = "out/common.yaml"
}

template {
  source               = "documents.ctmpl"
  destination          = "out/documents.json"
  perms                = 0600
}

4. Run your vault login ...
5. Run VAULT_TOKEN=$(cat ~/.vault-token) consul-template -config "templates.hcl"

[chilversc]

Yup, consul-template is the solution, its just not obvious and easy to miss when reading the documentation as it doesn't stand out as a standalone package. From the name sounds/feels like part of consul so is easy to ignore as "I'll worry about leaning and setting up consul later" when you're trying to focus on vault.

What would have helped me is when looking at https://www.vaultproject.io/docs/agent and the auto-auth methods it would mention that consul-template is stand-alone and can be used to automatically refresh a vault token.

[jovrum]

Stand-alone consul-template solves the use-case when providing secrets to applications using template sinks, but it doesn't solve the problem of local development when the application fetches secrets from an agent set up with auto-auth via Unix socket or local listener.

Supporting token auth in the agent would provide an obvious way for developers to use OIDC credentials during development, without changing application code.

edit:

I was able to work around this by having a companion AppRole for the OIDC role where the OIDC role is authorized to create single-use, short-lived secret IDs for the companion AppRole. A script then receives the OIDC session's auth token by environment variable, creates a secret ID and starts vault-agent with auto-auth through AppRole and the created secret ID. It's quite round-about and I would prefer if vault-agent could just start given a token directly, but at least this works.

[VioletHynes]

Hi there! Going to close this issue, as this feature has been implemented in #18740 and will be available in Vault 1.13.

Thanks for the report. The enthusiasm for this feature helped us prioritize it!