Vault:1.4.3 can not start with KMS auto unseal with IAM Role

[yevgeniyo-ps]

Describe the bug

With version 1.4.3 KMS auto unsealing with IAM Role is not working
Vault service is not starting at all, no output

/vault/logs # vault server -config=/vault/config/
^C

To Reproduce
Steps to reproduce the behavior:

1. Run vault 1.4.3 on docker with config:
   "seal": {
   "awskms": {
   "region": "your region",
   "kms_key_id": "your key"
   }
   }

2. Check logs

Vault server configuration file(s):

{
    "backend": {
      "mysql": {
        "address": "",
        "username": "",
        "password": "",
        "database": "",
        "ha_enabled": "true" 
      }
    },
    "listener": {
      "tcp":{
        "address": "0.0.0.0:8200",
        "tls_cert_file": "/vault/config/ssl/vault.crt",
        "tls_key_file": "/vault/config/ssl/vault.key"
      }
    },
    "ui": true,
    "log_level": "DEBUG",
    "seal": { 
      "awskms": {
        "region": "",
        "kms_key_id": ""
      }
    }
}

Additional context

Checked the same setup with 1.3.7
All works as expected

[kalafut]

Hi. Possibly the same as #8844? Note the comments at the end of that discussion.

[yevgeniyo-ps]

Looks like it has the same source

[danjbh]

I also encountered this issue and wasn't able to figure out what was going on until we set ui = false in the config, which then seemed to allow it to log some useful information.

Anyhow, my issue turned out to be a problem w/ the role -- I had it scoped to the wrong namespace.

Although I'd be interested in knowing if anyone else suddenly starts seeing error logs if they set ui = false. In the meantime, make sure your roles are setup correctly!

[tvoran]

@yevgeniyo Have you tried setting the environment variable AWS_ROLE_SESSION_NAME to something random in your docker container?

Otherwise you may be running into an issue with docker and IMDSv2:

• Get credentials from role has become slow aws/aws-sdk-go#2972 (comment)
• Provide an environment variable to disable IMDSv2 path aws/aws-sdk-go#2980
• AWS auth backend client unable to use IAM credentials from ECS task metadata #8847 (comment)

So you might try increasing the hop count for the underlying ec2 instances if you can.