Add IAM EC2 auth #161

maschwenk · 2017-12-14T21:35:21Z

I melded the gist that @tristanmorgan posted and #144. I also used this python snippet on the AWS documentation.

This uses the lightweight aws-sigv4 gem as suggested by @evanphx. The gist above had a method for achieving it completely without a gem but I noticed the gist had (what appeared to me to be) some errors:

It wasn't signing all the headers

It wasn't signing the body of the request:

k_date    = OpenSSL::HMAC.digest('sha256', 'AWS4' + credentials['SecretAccessKey'], date_stamp)
k_region  = OpenSSL::HMAC.digest('sha256', k_date, region_name)
k_service = OpenSSL::HMAC.digest('sha256', k_region, 'iam')
k_signing = OpenSSL::HMAC.digest('sha256', k_service, 'aws4_request')

...

"SignedHeaders=content-length;content-type;host;x-amz-date;x-vault-server, Signature=#{k_signing}"

whereas the ruby AWS SDK does and so does the python snippet:

 # Step 7: Combine elements to create create canonical request
 canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash

 algorithm = 'AWS4-HMAC-SHA256'
 credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request'
 string_to_sign = algorithm + '\n' +  amzdate + '\n' +  credential_scope + '\n' +  hashlib.sha256(canonical_request).hexdigest()

The basic outline for what I did is this:

Use the Instance Metadata API to retrieve the AWS secret key, access key, and region
Create an IAM signer for the STS endpoint
Sign request with the Vault headers and body included
Merge the Sigv4 headers generated by the gem with the Vault headers to create the payload for the Vault API

@evanphx There are a couple open questions on our side that I think you'd be in a good position to answer:

Does the Vault role name necessarily need to be the same as the AWS role name?
~~What headers are necessary in vault_headers? Do we need things like content-length?~~ In our testing it worked without it

TODO:

Tests (I don't see any testing for other auth methods so unsure how to proceed on that)
This is a fat method so I'm unsure if you'd like to split it up somehow

maschwenk · 2017-12-20T17:49:40Z

@evanphx What we're working on now is how this can work for both standard EC2 instances versus ECS tasks. ECS tasks use a different credentials API at:

"http://169.254.170.2#{ENV['AWS_CONTAINER_CREDENTIALS_RELATIVE_URI']}"

…te iam method

maschwenk · 2017-12-21T17:15:03Z

60f70e2 breaks the ECS/EC2 auth into two separate public methods that share an interface for iam

joelthompson · 2018-01-02T02:25:27Z

@maschwenk -- was thinking about this a bit. The Vault team understandably doesn't want to pull the entirety of the AWS SDK for Ruby into the Vault Ruby API client because a number of users probably won't even want it, and it could cause version conflicts with clients using a different version. And unfortunately, all the "magic" that AWS does on your behalf to figure out where to load credentials from isn't easily separable from the rest of the SDK. And the way that you're doing it won't allow clients to use explicit credentials, environment-variable-provided credentials, or AssumeRole-provided credentials (a workaround I suggest on the mailing list from time to time because I've been too lazy implement allowing binding multiple IAM principal ARNs to a single Vault role); doing any of those would require more code changes.

So, why not just move that little bit of complexity to the client, removing from client code the complexity of figuring out whether they have to call aws_ec2_iam or aws_ecs_iam? Have clients pass in a CredentialProvider object. Since Ruby is duck-typed, you don't need to pull in the SDK. If clients want the default credential chain, they just pass in Aws::CredentialProviderChain.new.resolve to get the default chain that the SDK uses (and that people are familiar with). If they want AssumeRole-supplied credentials, well, they just do exactly what the offical AWS documentation tells people to do and pass in the results of Aws::AssumeRoleCredentials.new

joelthompson · 2018-01-02T02:27:36Z

Also, @maschwenk, for future reference:

~~What headers are necessary in vault_headers? Do we need things like content-length?~~ In our testing it worked without it

Per the AWS documentation (see step 5), "The host header must be included as a signed header. If you include a date or x-amz-date header, you must also include that header in the list of signed headers."

maschwenk · 2018-01-04T21:45:00Z

@joelthompson sounds good. The only remaining piece in that puzzle is the region. Whereas previously we had it coming from the AWS Metadata API, we'd now need it to be statically passed in to the method, is that ok?

… api interaction

joelthompson · 2018-01-06T04:53:22Z

@maschwenk:

The only remaining piece in that puzzle is the region. Whereas previously we had it coming from the AWS Metadata API, we'd now need it to be statically passed in to the method, is that ok?

I'm... not sure. I think that, for now, the best way to handle this is actually to just have clients pass in the endpoint and then infer the region from the endpoint, with the endpoint defaulting to https://sts.amazonaws.com (and using the us-east-1 region in that case). It's not elegant, but it makes fewer assumptions about AWS and thus should be less likely to be buggy.

As to why I recommend this:

First, sorry this is such a long explanation, but this is a pretty complex topic that's not well documented.

STS is one of the weirdest AWS services -- it is "global" and has a single default global endpoint. But, unlike the other global services that I can think of off the top of my head (i.e., IAM and Route53), it has region-specific endpoints as well! AWS doesn't have great documentation about how to handle their home-spun authentication method in this case, so I did a bit of experimenting using boto3 (the python SDK, even though this is the Ruby SDK and Vault uses the golang SDK, and these are all corner cases that might not be well defined across SDKs) as it's the one I'm most familiar with.

Also an important note: the STS and IAM services are "global" but only within a "partition" and there are currently three partitions -- AWS Standard, GovCloud, and AWS China. It's unfortunately a bit tricky for users to test things out in one of the other partitions. Vault has some awkward code to try to handle this.

In this context, the region is needed for two reasons:

It's embedded in the 'credential scope" that is used in the authentication process (e.g., there's a string somewhere in the signing process that looks like 20180106/us-east-1/sts/aws4_request that is included in the authorization header and is covered by the signature)
It can be used to determine which endpoint to use; further, the endpoint actually gets embedded in the request via the Host header which, as I mentioned above, is one of the required headers.

The second point is a bit more subtle. The actual endpoint that the Vault server uses is determined solely by the Vault server and not at all by the client, defaulting to https://sts.amazonaws.com This is to prevent the Vault server from being abused by clients as a form of an open proxy. It looks like, when using the default endpoint of https://sts.amazonaws.com AWS expects a region in the credential scope of us-east-1.

So, if Vault admins do nothing, my proposed solution (defaulting to the endpoint of https://sts.amazonaws.com and defaulting to the region of us-east-1) should just work (I know, famous last words). And if the Vault server is configured to use a different STS endpoint, the clients need to know it to generate and sign the requests appropriately, and passing in the endpoint is how they would do it.

joelthompson · 2018-01-06T04:58:22Z

lib/vault/api/auth.rb

+      vault_headers = {
+        'User-Agent' => Vault::Client::USER_AGENT,
+        'Content-Type' => 'application/x-www-form-urlencoded; charset=utf-8',
+        'X-Vault-AWSIAM-Server-Id' => iam_auth_header_value


It's actually X-Vault-AWS-IAM-Server-ID (hyphen between AWS and IAM)

…ved, defaulting to https://sts.amazonaws.com (us-east-1)

maschwenk · 2018-01-08T18:45:58Z

@joelthompson Let me know if above commits follow your guidance above. Thanks so much for the thorough explanation. Maybe it makes sense to link to your PR/comment with regards to the STS endpoint.

joelthompson

Generally looking good to me :) The only thing I think really should be changed is the default server header value, and that should be a small change.

joelthompson · 2018-01-09T04:08:16Z

lib/vault/api/auth.rb

+      require "aws-sigv4"
+      require "base64"
+
+      valid_sts_endpoint = %r{https:\/\/sts.?(.*).amazonaws.com}.match(sts_endpoint)


FWIW, the STS endpoint for STS in the China (Beijing) region (cn-north-1) is sts.cn-north-1.amazonaws.com.cn. This will still work because your match doesn't have a $ at the end, but might be worth commenting so that future people don't add it and inadvertently break support for cn-north-1.

The endpoint for GovCloud is sts.us-gov-west-1.amazonaws.com so it should work exactly as intended.

joelthompson · 2018-01-09T04:11:09Z

lib/vault/api/auth.rb

@@ -13,6 +13,10 @@ def auth
  end

  class Authenticate < Request
+
+    # canary header used for aws_ec2_iam
+    IAM_SERVER_ID_HEADER = "canaryHeaderValue".freeze


I'm not a fan of this. Right now, it happens to work in Vault that if you haven't configured the server ID header value, it will accept any or no header value. This isn't great behavior, and might be something that I'd want to change in the future, so I'd rather not have the ruby client depend on it. Can you just make the default value nil and then just add the header value into the vault_headers if it's non-nil.

joelthompson · 2018-01-09T04:17:34Z

lib/vault/api/auth.rb

+        role: role,
+        iam_http_request_method: request_method,
+        iam_request_url: Base64.strict_encode64(sts_url),
+        iam_request_headers: Base64.strict_encode64(vault_headers.merge(sig4_headers).to_json),


This is probably OK, but just want to call out that this format for encoding the headers is only compatible with Vault starting with v0.8.0.

maschwenk · 2018-01-09T19:59:48Z

spec/integration/api/auth_spec.rb

+        expect(Secret).to receive(:decode).and_return secret
+        expect(::Aws::Sigv4::Signer).to(
+          receive(:new).with(
+            service: 'sts', region: 'cn-north-1', credentials_provider: credentials_provider


I think this expectation could obviate the need for the comment in e598ffc

joelthompson

Generally looks good to me from a Vault/AWS perspective! Thanks for doing this work! :)

I'm not too fluent in ruby, though, so I'm not the best person to comment on the details.

joelthompson · 2018-01-10T06:03:23Z

lib/vault/api/auth.rb

+
+      # STS in the China (Beijing) region (cn-north-1) is sts.cn-north-1.amazonaws.com.cn
+      # Take care changing below regex with that edge case in mind
+      valid_sts_endpoint = %r{https:\/\/sts.?(.*).amazonaws.com}.match(sts_endpoint)


I think you need to escape the . to be correct: https:\/\/sts\.?(.*)\.amazonaws\.com

joelthompson · 2018-01-10T06:06:07Z

spec/integration/api/auth_spec.rb

+
+    describe "#aws_iam" do
+      before(:context) do
+        vault_test_client.sys.enable_auth("aws", "aws", nil)


I think it would be good to also configure the header value here and then insert it, just to ensure it gets processed properly.

joelthompson · 2018-01-10T06:08:34Z

lib/vault/api/auth.rb

+    # @param [CredentialProvider] credentials_provider
+    #   https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/CredentialProvider.html
+    # @param [String] iam_auth_header_value optional
+    #   As of Jan 2018, Vault will accept ANY or NO header, but this is subject to change and should not be relied upon


Would be better to read: "Vault will accept ANY or NO header if none is configured by the Vault server admin"

maschwenk · 2018-01-25T00:37:19Z

@joelthompson Do you have anyone who would want to look at this in its updated state?

joelthompson

Hey @maschwenk -- one question and a couple cosmetic notes.

I do want to avoid any confusion, though. I'm not a HashiCorp employee, just a Vault community member and contributor who has done a fair amount of work on the AWS auth backend and who would like to see this merged because I think the community would benefit from your efforts on this PR. I jumped in hoping that doing so would make HashiCorp more willing to accept your PR but can't promise anything.

joelthompson · 2018-01-25T06:43:33Z

spec/integration/api/auth_spec.rb

@@ -216,6 +216,7 @@ module Vault
    describe "#aws_iam" do
      before(:context) do
        vault_test_client.sys.enable_auth("aws", "aws", nil)
+        vault_test_client.sys.put_auth_tune("aws", "iam_server_id_header_value" => "iam_header_canary")


I think the .sys.put_auth_tune here isn't right. The configuration of the header value is done via the auth/aws/config/client API endpoint, not via a mount-tune call.

joelthompson · 2018-01-25T06:44:03Z

lib/vault/api/auth.rb

+    # Take care changing below regex with that edge case in mind
+    #
+    # @param [String] sts_endpoint
+    #   The raw pem contents to use for the login procedure.


This looks like something left over from copy/pasting?

joelthompson · 2018-01-25T06:45:24Z

lib/vault/api/auth.rb

@@ -186,6 +186,60 @@ def aws_ec2(role, pkcs7, nonce = nil)
      return secret
    end

+    # Authenticate via IAM EC2 method by providing a AWS CredentialProvider (either ECS, AssumeRole, etc.)


It would be better to say something like "Authenticate via AWS IAM auth method" as this doesn't use the ec2 auth method at all.

flyinbutrs · 2018-02-02T19:47:05Z

@evanphx / @sethvargo - any updates on getting this merged in or what else needs to be done for it to be included?

sethvargo · 2018-02-02T20:17:09Z

I'm no longer employed at HashiCorp, so I can't help you. Sorry

flyinbutrs · 2018-02-02T20:21:12Z

Oops, thanks for responding anyway though!

maschwenk · 2018-02-02T20:37:33Z

@flyinbutrs @sethvargo I need to respond to Joel's good feedback sometime this weekend. We've had to monkeypatch this in our own projects to get it working. After I make the changes hopefully a Vault employee can review it

maschwenk · 2018-02-06T18:40:18Z

@flyinbutrs I've responded to Joel's feedback. I also need to do a little manual integration testing myself, in my current project I used the initial implementation I had here but have not swapped out for this final solution

maschwenk · 2018-02-11T03:20:00Z

Just did some integration testing on our own projects and this works excellently (currently just monkeypatching it in). It'd be great to get some feedback from a current employee of Hashicorp /cc @evanphx

maschwenk · 2018-02-22T18:44:13Z

@flyinbutrs Apologies, I'm hesitant to make any fixes until I get some indication that this might get merged....

flyinbutrs · 2018-02-22T19:52:46Z

No worries, understood. I'm using it off of your branch, and it's working great! Hopefully someone from hashicorp can take a look soon.

@briancain? I see you active from Hashicorp on other Ruby repos... Any chance you can poke the right person for vault-ruby? Or maybe @jefferai since you seem to be a maintainer on vault itself?

maschwenk · 2018-03-02T18:50:07Z

@evanphx I saw you merged a bunch of PR's on this repo this week. Any way you could cut a new release? I have a dependency on this in a gemspec so I don't have the option of pointing to master. Thanks!

maschwenk added 3 commits December 14, 2017 11:33

Add IAM EC2 auth

4432283

Service needs to be STS. Provide session token from credentials API

b70de61

Make Vault header optional

78c05e9

Split ECS/EC2 into seperate public methods delegating to single priva…

60f70e2

…te iam method

maschwenk mentioned this pull request Jan 4, 2018

Rubocop #162

Closed

Provide a CrendentialsProvider and region instead of complex metadata…

7602166

… api interaction

joelthompson reviewed Jan 6, 2018

View reviewed changes

maschwenk added 2 commits January 6, 2018 01:33

Fix header name

75fd1b0

Replace hard-coded region with STS endpoint from which region is deri…

cc1365b

…ved, defaulting to https://sts.amazonaws.com (us-east-1)

joelthompson mentioned this pull request Jan 8, 2018

AWS IAM auth 'Content-Length' header causes validation error when sent as number instead of string or []strings hashicorp/vault#3763

Closed

joelthompson reviewed Jan 9, 2018

View reviewed changes

maschwenk added 4 commits January 9, 2018 11:08

Remove canary header

128fdef

Add note about Beijing STS endpoint

e598ffc

Reorder params to have least likely to be overriden as the last

98ddc19

Correct wrong var name

11a5cde

maschwenk commented Jan 9, 2018

View reviewed changes

Add a mocked out integration test

6217a9f

maschwenk force-pushed the feature.add-ec2-iam-auth branch from e1871e8 to 6217a9f Compare January 9, 2018 20:56

joelthompson reviewed Jan 10, 2018

View reviewed changes

maschwenk added 4 commits January 10, 2018 11:43

Adjust documentation of header value

370d7e0

Escape the period because it's actually part of the url

d23372a

Split region method into seperate method so it can be easily unit tested

d8357d4

Formal doc

1321c11

maschwenk added 2 commits January 17, 2018 17:04

Add canary header to integration test

1a57422

Unit test for url parsing

9f4ff2f

joelthompson reviewed Jan 25, 2018

View reviewed changes

maschwenk added 2 commits February 3, 2018 22:16

Fix documentation

ca17bac

Properly test iam header check

43e211d

Fix documentation method name

63a54f7

evralston mentioned this pull request Feb 22, 2018

support auth_type GCP/GCE #165

Closed

evanphx merged commit ed2c063 into hashicorp:master Feb 27, 2018

evanphx mentioned this pull request Feb 27, 2018

Add support for aws auth backend for auth_type=iam #144

Closed

This was referenced Aug 29, 2018

auth_aws_iam method fails when using any region but the default us-east-1 hvac/hvac#251

Closed

RFC: First-Class AWS VPC Endpoint Support hashicorp/vault#5228

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add IAM EC2 auth #161

Add IAM EC2 auth #161

maschwenk commented Dec 14, 2017 •

edited

Loading

maschwenk commented Dec 20, 2017 •

edited

Loading

maschwenk commented Dec 21, 2017

joelthompson commented Jan 2, 2018

joelthompson commented Jan 2, 2018

maschwenk commented Jan 4, 2018 •

edited

Loading

joelthompson commented Jan 6, 2018

joelthompson Jan 6, 2018

maschwenk commented Jan 8, 2018

joelthompson left a comment

joelthompson Jan 9, 2018

joelthompson Jan 9, 2018

joelthompson Jan 9, 2018

maschwenk Jan 9, 2018

joelthompson left a comment

joelthompson Jan 10, 2018

joelthompson Jan 10, 2018

joelthompson Jan 10, 2018

maschwenk commented Jan 25, 2018

joelthompson left a comment

joelthompson Jan 25, 2018

joelthompson Jan 25, 2018

joelthompson Jan 25, 2018

flyinbutrs commented Feb 2, 2018

sethvargo commented Feb 2, 2018

flyinbutrs commented Feb 2, 2018

maschwenk commented Feb 2, 2018 •

edited

Loading

maschwenk commented Feb 6, 2018

maschwenk commented Feb 11, 2018 •

edited

Loading

maschwenk commented Feb 22, 2018

flyinbutrs commented Feb 22, 2018

maschwenk commented Mar 2, 2018

Add IAM EC2 auth #161

Add IAM EC2 auth #161

Conversation

maschwenk commented Dec 14, 2017 • edited Loading

TODO:

maschwenk commented Dec 20, 2017 • edited Loading

maschwenk commented Dec 21, 2017

joelthompson commented Jan 2, 2018

joelthompson commented Jan 2, 2018

maschwenk commented Jan 4, 2018 • edited Loading

joelthompson commented Jan 6, 2018

Choose a reason for hiding this comment

maschwenk commented Jan 8, 2018

joelthompson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joelthompson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maschwenk commented Jan 25, 2018

joelthompson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flyinbutrs commented Feb 2, 2018

sethvargo commented Feb 2, 2018

flyinbutrs commented Feb 2, 2018

maschwenk commented Feb 2, 2018 • edited Loading

maschwenk commented Feb 6, 2018

maschwenk commented Feb 11, 2018 • edited Loading

maschwenk commented Feb 22, 2018

flyinbutrs commented Feb 22, 2018

maschwenk commented Mar 2, 2018

maschwenk commented Dec 14, 2017 •

edited

Loading

maschwenk commented Dec 20, 2017 •

edited

Loading

maschwenk commented Jan 4, 2018 •

edited

Loading

maschwenk commented Feb 2, 2018 •

edited

Loading

maschwenk commented Feb 11, 2018 •

edited

Loading