Unable to Authenticate using AWS metadata credentials

[awilkins-frel]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

On an aws instance not in region with an IAM role assigned

1. Run vault login -method=aws role=dev-role
2. See error

Error authenticating: Error making API request.

URL: PUT http://localhost:8200/v1/auth/aws/login
Code: 400. Errors:

* error making upstream request: received error code 403 from STS: <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>SignatureDoesNotMatch</Code>
    <Message>Credential should be scoped to a valid region, not 'eu-west-1'. </Message>
  </Error>
  <RequestId>1e4e3999-cb3c-11e9-a3b7-9906bddec01b</RequestId>
</ErrorResponse>

Expected behavior
If everything else is in place - eg vault server has aws credentials, role exists in vault, iam role of instance configured for access to the role, login should succeed

Environment:

• Vault Server Version Version 1.2.2
• Vault CLI Version1.2.2
• Server Operating System/Architecture: Ubuntu 18.04.3 LTS amd64

Vault server configuration file(s):

ui = true

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault.d/staging-dc1-client.pem"
  tls_key_file  = "/etc/vault.d/staging-dc1-client-key.pem"
}`


seal "awskms" {
  region     = "eu-west-1"
  kms_key_id = "8f54xxxx-xxxx-xxxx-xxxx-xxxx0e8dcc85"
}


storage "consul" {
  session_ttl = "10s"
  lock_wait_time = "3s"
}

Additional context
A couple of intercepted packets between the cli and vault server seem to show what is going wrong here since 1.2.0

this is the iam_request_headers key base64 decoded

{"Authorization":["AWS4-HMAC-SHA256 Credential=ASIAYKKZPMH7PCTFWTXO/20190830/eu-west-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=529ae4b2741f13da4c9a599adbc24ba5eab9c03d510abdc39fc7209ed80b35a0"],"Content-Length":["43"],"Content-Type":["application/x-www-form-urlencoded; charset=utf-8"],"User-Agent":["aws-sdk-go/1.19.39 (go1.12.7; linux; amd64)"],"X-Amz-Date":["20190830T151437Z"],"X-Amz-Security-Token":["AgoJb3JpZ2luX2VjEP///////////wEaCWV1LXdlc3QtMSJHMEUCIB/C/AbOXQ/jPoJnecJ9ircLtvuhs1chYlqe0Q8G3oK2AiEArWiMqRX3QDGaONROEFRrrQ6IUEXHFK99gKgGK+HaR4sq4wMIqP//////////ARAAGgw1NzE5NTUxNzU5MzQiDEL5QYPfCYtGrIrpzyq3A37Mmv5ivuWnnL8TK6ZfvRoLFs8+jpMyb1NDXz29QvzxmnjoCDyXY2NAgdEjRbWWeOpSur5cqCLWY6hbsuVb4lGN5gsrBxM/bU+H3ny0mXRhbNpG6JFFix9oCoKw92kQt86RMHp4rK457hn0iOKD5IvyoR8MA1Weo7NPTqaaizB2x5uMKULaKz5q8oYwZQ6V8CKwcNEKUOC5/JhHgi9yQofuAS1rkli8vP2gmReSKODw4j09qZXs2fzk7F+ZSAqvKUS+843zfd0i7nFdp4LcBTag+drHJ/DPoqQu7v2ahql7elBkpl3I77OaDPs1ab9TG34eaeH5h6LtEjk5YpZO8uHdVoi22iv22u400iN2KRjltYGiMQ4j31aF1PLgE/lFV/GoiSHRrxuudPjMnKVx06KUjFs+k1F4t0mWjyV2VnUBqtxZU/k7qqqG8HHxgf4/SSEkdCcHpDMcp9xI1O3xYkizxT//nsxYm278PCyh/XJNd/XrtUIDyg+8zfS6N5MYyZ+xyoZ2UNHrXAGZwyCPT41JKQFykQGIPHO9FMy6NMHhxGS8S/fAAkEPKAeQfLJX0w9tajIEueUw5eKk6wU6tAE/FGoDifPvornMCTLPSlDwjayKc5UsmNPC8zOUj+UJAaROECMbpOJTUpl/5qURQvnrgyav7yjxPvsASC9dKbqEx/odR49bdnL+sz83vBVxJBOZdx3uS+2RE1AjP5MtOohcvOQxncBWlZJKZRgVHw15LsUw7DUWIZ56QX3xotHrrIqBWBC1Oe3o8G3qJnJoxkHLMnm8RyyEABJ9XlXa5tWgpCoyQOCoduVrIGvNyUtVWkjCMPE="]}

running it from cli version before 1.2.0 you see it look like this

{"Authorization":["AWS4-HMAC-SHA256 Credential=ASIAYKKZPMH7C76DYL5A/20190830/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=8eb25dc5b2030e07177c093d74c8be83a54b68be3e440b956d374a88dd8a2e27"],"Content-Length":["43"],"Content-Type":["application/x-www-form-urlencoded; charset=utf-8"],"User-Agent":["aws-sdk-go/1.15.55 (go1.11.5; linux; amd64)"],"X-Amz-Date":["20190830T140324Z"],"X-Amz-Security-Token":["AgoJb3JpZ2luX2VjEP7//////////wEaCWV1LXdlc3QtMSJIMEYCIQD5bZm7wsovV+/u3DysgZlwakKrV22UOz3rqYFsPdHDXgIhALFpSeDa7L9eDbX7HJlO84uPAUuvObMr9+V5rtyN1PcWKuMDCKf//////////wEQABoMNTcxOTU1MTc1OTM0IgxWdaLDWAoCItziX5oqtwO59OmzoctyQev+E1mLnJ5SEZ7eFlAphM3dwS/lDCPEz2goUv9Pu3Ou6EdSdIY01lzmofMuChyLciJWQfAQeaM17fI3+AcdWfQehY28OdmFjWaysGgqLPZ9ih5fgjoPKnHHriyZ7wKQoExVml6vFKzmZEpPBnr7LIMbJc+bsRfjh6MoKSPd1XfmE3L0j6GrxhcJ92pG6Gi6MFlOUACjO2Ak4vqNgH0NsVg2t8N4jww61HjrOYSQ1n/Jm0B+Wyfe+h75WNNO/VxeAkh+LudvCMgOa2k4sKPpz3D3lfHE2n8g1/chF40K8teP/uKaZuJBJ4PcqdDssu/Xf6VxnM8Cazo8Rntfvuiig063UIYTHbtPz0sQOmwAoOdNiR1uKxLEAHUFmP09ZU4ztpZKpjS7Ve5j3uu7ccfonEGve0eKQWq9GN0+IikqpEOUH+CuGAaItf6OuevAMi/cFSo7LH6LHb4iTFVNkPvst1qyZoiWI+JomtaHaTTVitbxHkwahd6BCzlHxqzoZz3Gby/O2ahAi4BUuiybCmyT3fsAT4/T6vSvZRu8ohUmjeo+EvciJYZP+/R64j78LJ7NMNXGpOsFOrMBjrc4wu6GjCZwxTzdJQHW+L8S8Iy5MH30p5F8OHTCVDrTTCpYIDcTyDQrvpLcOr1XUp4fpnWGDf1wOXtJZ3x1d9fPqJZ1SjJtEKkvFp+4oaqH5imgwLM/zV05Uvbf+J40LseP95r4L3uYycQBjyuWDuLItIPl4xsfP1F4tGdo17dUKhloqyoy2Jt4oI3/BLsDakF/OP7zWke/yCPkubo6EzAESf5YAIto4/scaQHgiZdp6fU="]}

In the older version the Credentials had the region of us-east-1 despite the instance being in eu-west-1

It looks to me like the sts end point needs these requests scoped to this region because that is the region it is in even if the instances are not.

[joelthompson]

Hi @awilkins-frel -- this change in behavior and how to handle it is described on the mailing list at https://groups.google.com/forum/#!msg/vault-tool/ki0GUEu7FFo/QBulNTDtBwAJ

[awilkins-frel]

Thanks that is a great help.
I was able to get things working by setting the STS Endpoint url on the vault server to the region we are using.
This will work fine for me because we are not using it in multiple regions, and if we do we'll probably use regional vault clusters.

What I am a little unclear of is if there is a way to make this work entirely client side at the moment using the vault CLI and vault agent.

[joelthompson]

You can simply set the AWS_REGION environment variable to us-east-1 or pass the region=us-east-1 parameter to the CLI's login command.

[awilkins-frel]

Thanks, I can get it to work on the CLI with the parameter, and can now see that has been updated in the docs
The agent was a little bit more complicated
I could not see any other way of getting it to work directly with the agent config, adding region to my auto_auth config did not work this is my config file

vault {
  address = "https://vault.service.consul:8200"
  tls_skip_verify = "true"
}


auto_auth {
  method "aws" {
    config = {
    type = "iam"
    role = "dev-role"
    }
  }
}


cache {
  use_auto_auth_token = true
}

listener "tcp" {
  address = "127.0.0.1:8100"
  tls_disable = "true"
}

adding
region = "us-east-1"
to auto auth method aws was ignored

I was able to get it to work using the environment variable you mentioned, but as it is running as a service I had to add it to systemd by adding
Environment=AWS_REGION=us-east-1
to the Service parameters.

I am still struggling slightly to get my head around the reason for this change.
Why would I want the client to by default auto detect the local regional end point and use it, if it will not work unless the vault server is also configured to use that same regional end point, which will not be the default?

Is there a step I am missing that would allow the server to auto detect the region the client is in, or for the client to pass on the region it has used for it's part of the authentication process?

[kalafut]

Fixed in #7622 and #7632

[mdshoaib707]

As per the vault doc, they say if sts_endpoint is set then sts_region should also be set.
Previously it was failing for me when I had only set sts_endpoint but later when sts_region was set then the error was gone.

Commands for setting the sts region and endpoint

vault write auth/aws/config/client sts_endpoint=https://sts.eu-west-1.amazonaws.com
vault write auth/aws/config/client sts_region=eu-west-1