Nomad Vault tokens generated from AWS Auth Engine fail to renew after upgrade to Vault 1.4.1

[angrycub]

This is not a Nomad issue, but can cause unexpected pain for Nomad users.

This is a mirror issue to hashicorp/vault#8972 for visibility; it should be monitored for work and solution. This will help folks that might search the Nomad issue base crosswalk back to the Vault issue.

---

Upgrading vault from 1.3.x to 1.4.x causes token renew-self failures with nomad 0.11.1 (and possibly others). This might be a silent BC break for nomad users. It went unnoticed for several days in my test environment, but I suspect that it could have lead to widespread downtime if left long enough.

Error message logged by nomad:

nomad.vault: got error or bad auth, so backing off: error="failed to renew the vault token: Error making API request.#012#012URL: PUT https://active.vault.service.consul:8200/v1/auth/token/renew-self#012Code: 500. Errors:#012#012* 1 error occurred:#012#011* failed to renew entry: unable to fetch instance ID from metadata during renewal#012#012" recoverable=true

Gitter discussions:

• https://gitter.im/hashicorp-nomad/Lobby?at=5eb99c3a8c044c1914946c21
• https://gitter.im/hashicorp-vault/Lobby?at=5eb9972d20eaac185302484d

The issue seems to be related to recent value changes related to EC2 instance metadata:

• Add helper for aliasmetadata and add to AWS auth vault#8783
• Document AWS auth metadata vault#8880

In my environment, adding back the the ec2_metadata attributes appears to completely fix the issue:

vault write auth/aws-ec2/config/identity ec2_metadata=account_id,auth_type,ami_id,instance_id,region iam_metadata=account_id,auth_type,canonical_arn,client_arn,client_user_id,inferred_aws_region,inferred_entity_id,inferred_entity_type

I also added iam_metadata based on a previous community member's findings, but those may not be necessary to fix this particular problem.

Environment:

• Vault Server Version (retrieve with vault status): 1.4.1
• Vault CLI Version (retrieve with vault version): 1.4.1
• Server Operating System/Architecture: CentOS 7

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.