Vault integration upgrade based on Workload Identity

[mikenomitch]

Proposal

Once Workload Identity upgrades make it into Nomad, we can redo the Vault integration to use these tokens as the source of auth instead of manually provided Vault tokens.

Using these tokens, Nomad Users would have a one-time set up process to integrate Nomad workloads into Vault.

The general flow for setting up the Vault-Nomad integration would be:

• Set up Vault
  • Create a Policy for Nomad in Vault
  • Enable the JWT Auth Method
  • Configure Vault to use Nomad’s Public keys - either passing in the keys, a JWKS URL, or an OIDC Config URL
  • Create a Vault Role for Nomad
• Set up Nomad
  • Pass a Vault URL into Nomad Server config in a new configuration block (or v2 of the existing vault block). (Note: no token needed)
• Deploy Job
  • Job is configured to use new Vault integration
  • Nomad, recognizing that the new integration is being used, automatically requests a token for this job using the JWT auth method.

This would involve an up front cost to set up roles in Vault, but after that no management of tokens would be needed.

Use Cases & Advantages

This would be advantageous in many ways:

• Nomad Users would not have to manage Vault token issuing, rotation, and revocation for Nomad clients.
• Nomad Users would not have to manage Vault tokens for each Nomad workload.
• Workload identity tokens could be time-bound and automatically rotated
• Workload identity tokens could provide fine-grained access at the task level
• Workload identity tokens could be automatically removed once the task has stopped
• Multiple Vault clusters could be configured to use the same token. This could allow Nomad to talk to multiple clusters at once, handle performance replicas better, and handle disaster recovery better. Failovers could happen without token rotation in Nomad.
• Vault could be more easily deployed as a Nomad job, as client tokens/config would not be needed.

Potential simultaneous improvements

While not directly related, there are a few other Vault improvements that should be considered while we do this upgrade:

• Supporting batch tokens in Vault
• Using Vault secrets in jobspecs, either in artifact stanza or Docker auth (I think this is likely unrelated, but such a popular feature that its worth considering while we implement)

[tgross]

Example of a motivating use case: #16639

[tgross]

Shipped in Nomad 1.7.0-beta.1

[tomqwpl]

Hi, would it be expected that to successfully retrieve the JWKS, you need to have verify_https_client set to "false"? If I have that set to true, it is expected that the client present a TLS certificate (mutual TLS), but Vault isn't going to do that, so the retrieval of JWKS fails?
Thanks.

[tgross]

@tomqwpl yeah that's right. We should probably update the verify_https_client docs to point out that it's safe to have verify_https_client=true for the API in general in the Nomad security model (so long as you've got ACLs enabled).

[tomqwpl]

that it's safe to have verify_https_client=true for the API in general

Do you mean "true" here? I'm assuming you mean "false"? I see in fact that the docs say "false".
I suspect we've enabled it because we're doing no further authentication to nomad, but I would need to clarify that.

Thanks.

[tgross]

Do you mean "true" here? I'm assuming you mean "false"? I see in fact that the docs say "false".

🤦 yes, sorry.

I suspect we've enabled it because we're doing no further authentication to nomad, but I would need to clarify that.

Yeah in that case you really so want to enable TLS verification for the HTTP API. But it's not recommended to have ACLs disabled.