Vault - AWS IAM/EC2 Auth- GovCloud

[marcuschaney]

All,

Has anyone experienced issues when attempting to generate credentials via Vault's "AWS Secrets Engine" for GovCloud? We're attempting to follow the documentation, but it seems to be an issue with the Region/Endpoints. https://www.vaultproject.io/docs/secrets/aws/index.html

The errors we receive are below:
vault login -method=aws role=test-role-iam

Error authenticating: Error making API request.

URL: PUT http://XXXXXXXX:8200/v1/auth/aws/login
Code: 400. Errors:

• error making upstream request: received error code 403 from STS:
  
  Sender
  SignatureDoesNotMatch
  Credential should be scoped to a valid region, not 'us-east-1'.
  
  XXXXXXXXXXXX

Also, we noticed that there isn't a drop down selection in the GUI for "us-gov-west-1", so is this even being read when we pass in the following argument "vault write aws/config/root \ region=us-gov-west-1" ?

[marcuschaney]

The URL it's attempting to access is https://sts.amazonaws.com and default to us-east-1

[joelthompson]

Hi @marcuschaney -- a couple things going on here.

First, you mention the AWS Secrets Engine, but it looks like you're actually trying to use the AWS auth method, correct?

Assuming that's the case, you need to set the sts_endpoint parameter in the auth/aws/config/client endpoint to the STS endpoint in the GovCloud region you are using. Given the error message, it also looks like you have done that? If not, you'll also need to do this. Note that writing to aws/config/root will configure the AWS Secrets Engine, which is configured separately from the AWS auth method.

Third, you need to ensure the Vault client CLI is configured to send login data for the correct AWS region. This is a bit... tricky. I think that GovCloud operates a bit differently than the AWS standard partition when it comes to STS, but it's hard for me to verify that personally. I suggest you try running export AWS_REGION=us-gov-west-1 first (or us-gov-east-1, depending on whichever region you want to use STS in) and then try running vault login -method=aws role=test-role-iam and seeing if that works.

[marcuschaney]

@joelthompson Awesome, thank you so much! After exporting the region (even though it was set in AWS configure), my colleague was able to successfully generate an auth token. So he is unblocked!

For me, I assume the resolution would be similar? I'm actually trying to utilize the Secrets Engine. Will test that this afternoon.

[joelthompson]

@joelthompson Awesome, thank you so much! After exporting the region (even though it was set in AWS configure), my colleague was able to successfully generate an auth token. So he is unblocked!

Great! Unfortunately you're running into one of the maddening inconsistencies between the different languages' SDKs. I think that you could also set the AWS_SDK_LOAD_CONFIG environment variable to coax the SDK to read the configuration out of the config files.

For me, I assume the resolution would be similar? I'm actually trying to utilize the Secrets Engine. Will test that this afternoon.

I'm not as familiar with the AWS Secrets Engine as I am with the auth method. I think configuring the region as you mentioned should just work as you expected, but if not, you may also need to configure the sts_endpoint parameter in the aws/config/root endpoint (and possibly also iam_endpoint) for the secrets engine. If you run into problems, feel free to post your errors here and I'll try to help you out (or post to the mailing list).

[marcuschaney]

@joelthompson looking good with the aws auth in GovCloud! I actually don't need to use the AWS Secret engine, that was just miscommunication with management. So now that I'm working in GovCloud, I've run into an odd issue in a different partition.

After doing the AWS_REGION=us-test-east-1 export and writing the STS, IAM and EC2 endpoints to auth/aws/config/client, I still receive the below error:

"unable to resolve partition "aws-test-b" (which is oddly how the ARNs are identified is this partition) to a region.

Any ideas?

[joelthompson]

Hi @marcuschaney -- I know why Vault is doing that, but I'm a bit confused as to why you have ARNs with a partition of aws-test-b and a region of us-test-east-1.

Basically, you're running into this bit of code that was intended to try to support GovCloud and the China partition, hopefully the comment explains what was going on:

vault/builtin/credential/aws/backend.go

Lines 221 to 235 in 4cba25c

	// This odd-looking code is here because IAM is an inherently global service. IAM and STS ARNs
	// don't have regions in them, and there is only a single global endpoint for IAM; see
	// http://docs.aws.amazon.com/general/latest/gr/rande.html#iam_region
	// However, the ARNs do have a partition in them, because the GovCloud and China partitions DO
	// have their own separate endpoints, and the partition is encoded in the ARN. If Amazon's Go SDK
	// would allow us to pass a partition back to the IAM client, it would be much simpler. But it
	// doesn't appear that's possible, so in order to properly support GovCloud and China, we do a
	// circular dance of extracting the partition from the ARN, finding any arbitrary region in the
	// partition, and passing that region back back to the SDK, so that the SDK can figure out the
	// proper partition from the arbitrary region we passed in to look up the endpoint.
	// Sigh
	region := getAnyRegionForAwsPartition(entity.Partition)
	if region == nil {
	return "", fmt.Errorf("unable to resolve partition %q to a region", entity.Partition)
	}

In order for it to work, you need to have a partition that's known to the AWS golang SDK. According to https://docs.aws.amazon.com/govcloud-us/latest/ug-west/using-govcloud-arns.html GovCloud ARNs should have a partition of aws-us-gov and not aws-test-b.

[marcuschaney]

@joelthompson

Hey, thanks for the response! So this is actually a classified partition, that’s why I submitted the name with “test”. But yeah, isn’t odd that the partition is prefixed with “aws” but the actual region is prefixed with “us”. This environment is always a pain, so I figure I’ll have to implement some kind of workaround. I can successfully curl the STS and IAM endpoints, but when Vault calls IAM for the role, it comes back with that error.

[marcuschaney]

@joelthompson so I guess I can actually divulge the partition and region. Please see below:

aws-iso-b is the partition and the region is us-isob-east-1

[marcuschaney]

@joelthompson RESOLVED! So, instead of specifying IAM Principal like below:

vault write auth/aws/role/feds-role-iam
auth_type=iam
bound_iam_principal_arn=arn:aws-us-gov:iam::xxxxxxxxxx:role/my-EC2Role
max_ttl=1h
policies=my-secrets-policy.hcl

---

I specified IAM Instance Profile instead, then inferred EC2 instance type along with the relevant region.

vault write auth/aws/role/feds-role-iam
auth_type=iam
bound_iam_instance_profile_arn=arn:aws-us-gov:iam::xxxxxxxxxxx:role/my-EC2Role
max_ttl=1h
policies=my-secrets-policy.hcl
inferred_entity_type=ec2_instance
inferred_aws_region=us-isob-b-1

[marcuschaney]

@joelthompson so I got passed that just to run into the 403 error from STS attempting to reach us-east-1, even after doing "export AWS_REGION=us-isob-east-1. Also, I ensure that was set via "aws configure".

I saw here that fix was implemented to specify region at login, what version was that implemented in?

99bd2af

I'm running v1.0.1

[joelthompson]

Hi @marcuschaney -- I don't believe that fix has made it into a Vault release yet.

Anyway, some general thoughts:

1. I'm not sure your trick of putting the aws-us-gov partition in the instance profile will actually work when it comes time to login -- it'll likely fail out when trying to match the role to the instance profile.
2. The Vault CLI is really just a helper to generate the headers in the format that the SDK expects them. See https://gist.github.com/krish7919/44d56dd8a77f3dee94d06b6f1662a8b4 for a bash implementation to produce many of the same headers that's more configurable. I'd suggest using and modifying that to generate the login data as it will give you more control over the region.
3. Your best bet might be to try setting resolve_aws_unique_ids to false in your role (but see the security note in the documentation!) and keep your original bound_iam_principal_arn.

I'll try to think some more about how best to support this kind of use case.

[marcuschaney]

Hey @joelthompson I've decided to go the EC2 auth path, which doesn't required the STS call. Now, I'm getting a "500 Internal Server Error" when attempting to do the following:

pkcs7=$(curl -s
"http://169.254.169.254/latest/dynamic/instance-identity/pkcs7" | tr -d '\n')
data=$(cat <<EOF
{
"role": "my-role",
"pkcs7": "$pkcs7"
}
EOF
)

curl --request POST
--fail
--data "$data"
"https://vault.domain/v1/auth/aws/login"

Same when I go directly to the server IP via port 8200.

[marcuschaney]

@joelthompson trying the workaround you mentioned here: #4014

$ vault write auth/aws/login identity=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | base64 -w 0) signature=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/signature | tr -d '\n') role=<ROLE_NAME>

I now get the below error:
{"errors":["instance identity verification using SHA256 RSA signature is unsuccessful"]}

This integration is kicking my butt! I appreciate all your help!

[joelthompson]

Hey @marcuschaney -- just a heads up, you might run into similar issues when Vault tries to make AWS EC2 API calls to validate that the instance is real.

Anyway, in this case, you need to teach Vault about the certificate that it should use to verify the signature, as described at https://www.vaultproject.io/docs/auth/aws.html#varying-public-certificates -- you'll likely need to talk to your AWS support contacts to get the public key.

[joelthompson]

Also, just thinking a bit outside the box, does AWS provide SDKs for the classified region? If so, maybe you'll have better luck just forking Vault and swapping out the AWS golang SDK with the alternate SDK?

[marcuschaney]

@joelthompson Perfect, thank you for that info! I'll reach out to support for the cert. Also, I'll inquire about the SDKs for that region, but I know we've encountered issues dealing with SDKs up there before.

I'll update accordingly. Thanks!

[marcuschaney]

@joelthompson was able to get the cert from AWS! Have you ever seen this error when attempting to write to Vault's certificate directory:

Code 500. Errors:

1 error occurred:
* invalid certificate; should be on PEM block only

I've tried this without the "---Begin" etc, and without that. Looking at the documentation, it just shows the cert without the "Begin" and "End"; however, I still get this funky error.

I'm trying "vault write auth/aws/config/certificate/awspublic aws_public_cert=awspublic

[joelthompson]

Hi @marcuschaney -- actually, you need to base64-encode the certificate (including the beginning "-----BEGIN..." and ending "-----END..." lines). If you look at the docs (https://www.vaultproject.io/api/auth/aws/index.html#sample-payload-2) they show a sample payload of LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM3VENDQXEwQ0NRQ1d1a2paNVY0YVp6QUpCZ2NxaGtqT09BUURNRnd4Q3pBSkJnTlZCQVlUQWxWVE1Sa3cKRndZRFZRUUlFeEJYWVhOb2FXNW5kRzl1SUZOMFlYUmxNUkF3RGdZRFZRUUhFd2RUWldGMGRHeGxNU0F3SGdZRApWUVFLRXhkQmJXRjZiMjRnVjJWaUlGTmxjblpwWTJWeklFeE1RekFlRncweE1qQXhNRFV4TWpVMk1USmFGdzB6Ck9EQXhNRFV4TWpVMk1USmFNRnd4Q3pBSkJnTlZCQVlUQWxWVE1Sa3dGd1lEVlFRSUV4QlhZWE5vYVc1bmRHOXUKSUZOMFlYUmxNUkF3RGdZRFZRUUhFd2RUWldGMGRHeGxNU0F3SGdZRFZRUUtFeGRCYldGNmIyNGdWMlZpSUZObApjblpwWTJWeklFeE1RekNDQWJjd2dnRXNCZ2NxaGtqT09BUUJNSUlCSHdLQmdRQ2prdmNTMmJiMVZRNHl0LzVlCmloNU9PNmtLL24xTHpsbHI3RDhad3RRUDhmT0VwcDVFMm5nK0Q2VWQxWjFnWWlwcjU4S2ozbnNzU05wSTZiWDMKVnlJUXpLN3dMY2xuZC9Zb3pxTk5tZ0l5WmVjTjdFZ2xLOUlUSEpMUCt4OEZ0VXB0M1FieVlYSmRtVk1lZ042UApodmlZdDVKSC9uWWw0aGgzUGExSEpkc2tnUUlWQUxWSjNFUjExK0tvNHRQNm53dkh3aDYrRVJZUkFvR0JBSTFqCmsrdGtxTVZIdUFGY3ZBR0tvY1Rnc2pKZW02LzVxb216SnVLRG1iSk51OVF4dzNyQW90WGF1OFFlK01CY0psL1UKaGh5MUtIVnBDR2w5ZnVlUTJzNklMMENhTy9idXljVTFDaVlRazQwS05IQ2NIZk5pWmJkbHgxRTlycFVwN2JuRgpsUmEydjFudE1YM2NhUlZEZGJ0UEVXbWR4U0NZc1lGRGs0bVpyT0xCQTRHRUFBS0JnRWJtZXZlNWY4TElFL0dmCk1ObVA5Q001ZW92UU9HeDVobzhXcUQrYVRlYnMrazJ0bjkyQkJQcWVacXBXUmE1UC8ranJkS21sMXF4NGxsSFcKTVhyczNJZ0liNitoVUlCK1M4ZHo4L21tTzBicHI3NlJvWlZDWFlhYjJDWmVkRnV0N3FjM1dVSDkrRVVBSDVtdwp2U2VEQ09VTVlRUjdSOUxJTll3b3VISXppcVFZTUFrR0J5cUdTTTQ0QkFNREx3QXdMQUlVV1hCbGs0MHhUd1N3CjdIWDMyTXhYWXJ1c2U5QUNGQk5HbWRYMlpCclZOR3JOOU4yZjZST2swazlLCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K

When base64-decoded, that becomes:

-----BEGIN CERTIFICATE-----
MIIC7TCCAq0CCQCWukjZ5V4aZzAJBgcqhkjOOAQDMFwxCzAJBgNVBAYTAlVTMRkw
FwYDVQQIExBXYXNoaW5ndG9uIFN0YXRlMRAwDgYDVQQHEwdTZWF0dGxlMSAwHgYD
VQQKExdBbWF6b24gV2ViIFNlcnZpY2VzIExMQzAeFw0xMjAxMDUxMjU2MTJaFw0z
ODAxMDUxMjU2MTJaMFwxCzAJBgNVBAYTAlVTMRkwFwYDVQQIExBXYXNoaW5ndG9u
IFN0YXRlMRAwDgYDVQQHEwdTZWF0dGxlMSAwHgYDVQQKExdBbWF6b24gV2ViIFNl
cnZpY2VzIExMQzCCAbcwggEsBgcqhkjOOAQBMIIBHwKBgQCjkvcS2bb1VQ4yt/5e
ih5OO6kK/n1Lzllr7D8ZwtQP8fOEpp5E2ng+D6Ud1Z1gYipr58Kj3nssSNpI6bX3
VyIQzK7wLclnd/YozqNNmgIyZecN7EglK9ITHJLP+x8FtUpt3QbyYXJdmVMegN6P
hviYt5JH/nYl4hh3Pa1HJdskgQIVALVJ3ER11+Ko4tP6nwvHwh6+ERYRAoGBAI1j
k+tkqMVHuAFcvAGKocTgsjJem6/5qomzJuKDmbJNu9Qxw3rAotXau8Qe+MBcJl/U
hhy1KHVpCGl9fueQ2s6IL0CaO/buycU1CiYQk40KNHCcHfNiZbdlx1E9rpUp7bnF
lRa2v1ntMX3caRVDdbtPEWmdxSCYsYFDk4mZrOLBA4GEAAKBgEbmeve5f8LIE/Gf
MNmP9CM5eovQOGx5ho8WqD+aTebs+k2tn92BBPqeZqpWRa5P/+jrdKml1qx4llHW
MXrs3IgIb6+hUIB+S8dz8/mmO0bpr76RoZVCXYab2CZedFut7qc3WUH9+EUAH5mw
vSeDCOUMYQR7R9LINYwouHIziqQYMAkGByqGSM44BAMDLwAwLAIUWXBlk40xTwSw
7HX32MxXYruse9ACFBNGmdX2ZBrVNGrN9N2f6ROk0k9K
-----END CERTIFICATE-----

[marcuschaney]

@joelthompson Awesome, that did it!! Thank you so much for all your assistance! I was able to successfully add the cert (even though curl -v showed "no content" during the PUT). I was able to verify by doing:

curl
--header "X-Vault-Token: $vault_token"
http://vault.domain/v1/auth/aws/config/certificate/aws-public-cert

Which returned the decoded version of the certificate along with a request ID.