[1.4.x] Nomad integration breakage with aws / aws-ec2 auth and instance metadata

[stevenscg]

Upgrading vault from 1.3.x to 1.4.x causes token renew-self failures with nomad 0.11.1 (and possibly others). This might be a silent BC break for nomad users. It went unnoticed for several days in my test environment, but I suspect that it could have lead to widespread downtime if left long enough.

Error message logged by nomad:

nomad.vault: got error or bad auth, so backing off: error="failed to renew the vault token: Error making API request.#012#012URL: PUT https://active.vault.service.consul:8200/v1/auth/token/renew-self#012Code: 500. Errors:#012#012* 1 error occurred:#012#011* failed to renew entry: unable to fetch instance ID from metadata during renewal#012#012" recoverable=true

Gitter discussions:

• https://gitter.im/hashicorp-nomad/Lobby?at=5eb99c3a8c044c1914946c21
• https://gitter.im/hashicorp-vault/Lobby?at=5eb9972d20eaac185302484d

The issue seems to be related to recent value changes related to EC2 instance metadata:

• Add helper for aliasmetadata and add to AWS auth #8783
• Document AWS auth metadata #8880

In my environment, adding back the the ec2_metadata attributes appears to completely fix the issue:

vault write auth/aws-ec2/config/identity ec2_metadata=account_id,auth_type,ami_id,instance_id,region iam_metadata=account_id,auth_type,canonical_arn,client_arn,client_user_id,inferred_aws_region,inferred_entity_id,inferred_entity_type

I also added iam_metadata based on a previous community member's findings, but those may not be necessary to fix this particular problem.

Environment:

• Vault Server Version (retrieve with vault status): 1.4.1
• Vault CLI Version (retrieve with vault version): 1.4.1
• Server Operating System/Architecture: CentOS 7

[stevenscg]

The upgrade guide is helpful here. I had missed it originally. As a user (vault and/or nomad), I think I would like to know which of ec2_metadata and iam_metadata values are actually required for proper operation so I can trim the values to only those that are required.

[avoidik]

I faced the same, in some cases mandatory metadata field will be required

https://github.com/hashicorp/vault/blob/master/builtin/credential/aws/path_login.go#L965-L969
https://github.com/hashicorp/vault/blob/master/builtin/credential/aws/path_login.go#L1073-L1077

[ianferguson]

We ran into this issue upgrading from v1.3.4 to 1.4.1 (not using Nomad).

While the upgrade guide did note the change in behavior, it did not note that the canonical_arn attribute was required in order to renew tokens issued using AWS IAM (seen in the code snippet @avoidik shared above), which caused our consumers' renewal processes fail, leading to a spike in login attempts until we configured the auth/aws/config/identity configuration to retain attributes again.

[kalafut]

The update to the metadata handling was indeed an API change, but it definitely should not have broken token renewal. A fix will be coming soon, and in the interim the manual configuration you've mentioned is the correct mitigation. We've added a 1.4.1 upgrade guide and updated the 1.3.5 guide to describe this situation and workaround.

After the fix, the metadata customization will not be a requirement. You'll be able to set that as appropriate (or not at all), and Vault will always retain what is needed for token renewal.

Thank you for reporting this!