IMDSv2 not supported #243

burbma · 2023-10-10T03:04:06Z

Amazon Linux 2023 does not allow use of IMDSv1 but rather requires use of IMDSv2 (see here for the difference). cognitect.aws.ec2-metadata-utils only allows use of v1. This means aws-api cannot authenticate with Instance Profile Credentials (number 6 in the credential provider chain, as of this writing, here).

Dependencies

{:deps {com.cognitect.aws/api {:mvn/version "0.8.686"}}}

Description with failing test case

First, be on an ec2 running Amazon Linux 2023. Then,

(require '[cognitect.aws.ec2-metadata-utils :refer :all])
(require '[cognitect.aws.http.cognitect :as http])
(def http-client (http/create))
(get-ec2-instance-data http-client)
;; See that this returns nil

There are two problems here. First, it didn't return instance metadata. Second, it returned nil rather than failing. If you look this line you'll see it's swallowing any errors from this request. When I run the http request from these lines manually to inspect the response I see this following swallowed error

{:status 401, :headers {"server" "EC2ws", "connection" "close", "content-length" "0", "date" "Sat, 07 Oct 2023 02:59:57 GMT", "content-type" "text/plain"}, :body nil, :cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.anomalies/message "HTTP protocol violation: Authentication challenge without WWW-Authenticate header", :cognitect.http-client/throwable #error {
 :cause "HTTP protocol violation: Authentication challenge without WWW-Authenticate header"
 :via
 [{:type org.eclipse.jetty.client.HttpResponseException
   :message "HTTP protocol violation: Authentication challenge without WWW-Authenticate header"
   :at [org.eclipse.jetty.client.AuthenticationProtocolHandler$AuthenticationListener onComplete "AuthenticationProtocolHandler.java" 164]}]
 :trace
 [[org.eclipse.jetty.client.AuthenticationProtocolHandler$AuthenticationListener onComplete "AuthenticationProtocolHandler.java" 164]
  [org.eclipse.jetty.client.ResponseNotifier notifyComplete "ResponseNotifier.java" 218]
  [org.eclipse.jetty.client.ResponseNotifier notifyComplete "ResponseNotifier.java" 210]
  [org.eclipse.jetty.client.HttpReceiver terminateResponse "HttpReceiver.java" 481]
  [org.eclipse.jetty.client.HttpReceiver terminateResponse "HttpReceiver.java" 461]
  [org.eclipse.jetty.client.HttpReceiver responseSuccess "HttpReceiver.java" 424]
  [org.eclipse.jetty.client.http.HttpReceiverOverHTTP messageComplete "HttpReceiverOverHTTP.java" 374]
  [org.eclipse.jetty.http.HttpParser handleContentMessage "HttpParser.java" 597]
  [org.eclipse.jetty.http.HttpParser parseContent "HttpParser.java" 1668]
  [org.eclipse.jetty.http.HttpParser parseNext "HttpParser.java" 1551]
  [org.eclipse.jetty.client.http.HttpReceiverOverHTTP parse "HttpReceiverOverHTTP.java" 208]
  [org.eclipse.jetty.client.http.HttpReceiverOverHTTP process "HttpReceiverOverHTTP.java" 148]
  [org.eclipse.jetty.client.http.HttpReceiverOverHTTP receive "HttpReceiverOverHTTP.java" 80]
  [org.eclipse.jetty.client.http.HttpChannelOverHTTP receive "HttpChannelOverHTTP.java" 131]
  [org.eclipse.jetty.client.http.HttpConnectionOverHTTP onFillable "HttpConnectionOverHTTP.java" 172]
  [org.eclipse.jetty.io.AbstractConnection$ReadCallback succeeded "AbstractConnection.java" 311]
  [org.eclipse.jetty.io.FillInterest fillable "FillInterest.java" 105]
  [org.eclipse.jetty.io.ChannelEndPoint$1 run "ChannelEndPoint.java" 104]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill runTask "EatWhatYouKill.java" 338]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill doProduce "EatWhatYouKill.java" 315]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill tryProduce "EatWhatYouKill.java" 173]
  [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill produce "EatWhatYouKill.java" 137]
  [org.eclipse.jetty.util.thread.QueuedThreadPool runJob "QueuedThreadPool.java" 883]
  [org.eclipse.jetty.util.thread.QueuedThreadPool$Runner run "QueuedThreadPool.java" 1034]
  [java.lang.Thread run "Thread.java" 833]]}}

Indeed IMDSv2 requires a form of authentication that v1 does not.

The text was updated successfully, but these errors were encountered:

scottbale · 2023-10-12T21:11:39Z

Thanks for the very thorough writeup!

tjg · 2023-11-05T02:44:33Z

Hi! For anyone curious, here's how I got it working under IMDSv2

Maybe someting like it could be mentioned in the README, so people don't have to stub their toes on this problem?

I'd be grateful for any ruthless criticism. Like:

Should I use retries?
How to handle failures in CredentialsProvider's .fetch method? (Return nil, throw an exception, etc?)

Soooo, I can declare:

(def s3
  (aws/client {:api :s3
               :region "us-east-1"
               :credentials-provider (role-credentials-provider)}))

Using:

(ns test-test-test.aws-api-auth
  (:require
   [clj-http.client :as client]
   [clojure.data.json :as json]
   [cognitect.aws.client.api :as aws]
   [cognitect.aws.credentials :as credentials]))

(def ^:private token-url
  "http://169.254.169.254/latest/api/token")

(def ^:private security-credentials-url
  "http://169.254.169.254/latest/meta-data/iam/security-credentials/")

(defn- get-imdsv2-token []
  (let [{:keys [body status] :as response}
        (client/put token-url
                    {:headers {"X-aws-ec2-metadata-token-ttl-seconds" 
                               "21600"}
                     :throw-exceptions false})]
    (if (= 200 status)
      body
      (throw (ex-info "No IMDSv2 token available." 
                      {:IMDSv2-status-code status})))))

(defn- get-iam-role-name [token]
  (let [{:keys [body status] :as response}
        (client/get security-credentials-url
                    {:headers {"X-aws-ec2-metadata-token" token}
                     :throw-exceptions false})]
    (if (= 200 status)
      body
      (throw (ex-info "No IAM role found." 
                      {:IMDSv2-status-code status})))))

(defn- get-iam-role-credentials
  ([token]
   (let [role-name (get-iam-role-name token)]
     (when role-name
       (get-iam-role-credentials token role-name))))
  ([token role-name]
   (let [{:keys [body status] :as response}
         (client/get (str security-credentials-url role-name)
                     {:headers {"X-aws-ec2-metadata-token" token}
                      :throw-exceptions false})]
     (if (= 200 status)
       (json/read-str body :key-fn keyword)
       (throw (ex-info "No IAM role credentials found." 
                       {:IMDSv2-status-code status}))))))

(defn- get-credentials []
  (some->> (get-imdsv2-token)
           (get-iam-role-credentials)))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; Public API

(defn role-credentials-provider []
  (credentials/cached-credentials-with-auto-refresh
   (reify credentials/CredentialsProvider
     (fetch [_]
       (when-let [creds (try
                          (get-credentials)
                          ;; Err... what happens if I throw in here?
                          ;; Or return nil?
                          (catch Exception e))]
         {:aws/access-key-id     (:AccessKeyId creds)
          :aws/secret-access-key (:SecretAccessKey creds)
          :aws/session-token     (:Token creds)
          ::credentials/ttl      (credentials/calculate-ttl creds)})))))

brandonstubbs · 2024-01-12T17:18:24Z

Hi

It seems there is another ticket requesting IMDSv2 support #165

While Amazon Linux 2023 uses IMDSv2 by default (Which you should use as best practice) you can still change this.
https://docs.aws.amazon.com/linux/al2023/ug/compare-with-al2.html#imdsv2

For example, launching a new instance via cli

aws ec2 run-instances \
	...
    --metadata-options "HttpTokens=optional"

Or via the console in Launch an instance (Advanced details):

tlonist-sang · 2024-03-06T00:08:24Z

Hi! For anyone curious, here's how I got it working under IMDSv2

Maybe someting like it could be mentioned in the README, so people don't have to stub their toes on this problem?

I'd be grateful for any ruthless criticism. Like:

Should I use retries?
How to handle failures in CredentialsProvider's .fetch method? (Return nil, throw an exception, etc?)

Soooo, I can declare:

(def s3
  (aws/client {:api :s3
               :region "us-east-1"
               :credentials-provider (role-credentials-provider)}))

Using:

(ns test-test-test.aws-api-auth
  (:require
   [clj-http.client :as client]
   [clojure.data.json :as json]
   [cognitect.aws.client.api :as aws]
   [cognitect.aws.credentials :as credentials]))

(def ^:private token-url
  "http://169.254.169.254/latest/api/token")

(def ^:private security-credentials-url
  "http://169.254.169.254/latest/meta-data/iam/security-credentials/")

(defn- get-imdsv2-token []
  (let [{:keys [body status] :as response}
        (client/put token-url
                    {:headers {"X-aws-ec2-metadata-token-ttl-seconds" 
                               "21600"}
                     :throw-exceptions false})]
    (if (= 200 status)
      body
      (throw (ex-info "No IMDSv2 token available." 
                      {:IMDSv2-status-code status})))))

(defn- get-iam-role-name [token]
  (let [{:keys [body status] :as response}
        (client/get security-credentials-url
                    {:headers {"X-aws-ec2-metadata-token" token}
                     :throw-exceptions false})]
    (if (= 200 status)
      body
      (throw (ex-info "No IAM role found." 
                      {:IMDSv2-status-code status})))))

(defn- get-iam-role-credentials
  ([token]
   (let [role-name (get-iam-role-name token)]
     (when role-name
       (get-iam-role-credentials token role-name))))
  ([token role-name]
   (let [{:keys [body status] :as response}
         (client/get (str security-credentials-url role-name)
                     {:headers {"X-aws-ec2-metadata-token" token}
                      :throw-exceptions false})]
     (if (= 200 status)
       (json/read-str body :key-fn keyword)
       (throw (ex-info "No IAM role credentials found." 
                       {:IMDSv2-status-code status}))))))

(defn- get-credentials []
  (some->> (get-imdsv2-token)
           (get-iam-role-credentials)))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; Public API

(defn role-credentials-provider []
  (credentials/cached-credentials-with-auto-refresh
   (reify credentials/CredentialsProvider
     (fetch [_]
       (when-let [creds (try
                          (get-credentials)
                          ;; Err... what happens if I throw in here?
                          ;; Or return nil?
                          (catch Exception e))]
         {:aws/access-key-id     (:AccessKeyId creds)
          :aws/secret-access-key (:SecretAccessKey creds)
          :aws/session-token     (:Token creds)
          ::credentials/ttl      (credentials/calculate-ttl creds)})))))

Thanks for sharing your code!
I think returning nil makes more sense here, especially when you want to include this provider in chain-credentials-provider. Inside of it is a function called valid-credentials that checks if the fetched credentials is valid. The creds having nil will be handled by the function.

dharrigan · 2024-03-11T12:36:48Z

This bit me hard today on a production system (causing production issues). Setting it to be "optional" in the console got round this problem (for now! - we have an automated deployment system, so have to now figure out how to shove this into the metadata setup of the ec2 instance).

Please can support for IMDSv2 within the codebase be considered as soon as possible.

Thank you.

-=david=-

zerg000000 · 2024-03-11T13:15:55Z

I think the region provider has a similar issue, but it should have its own ticket.

scottbale · 2024-10-16T19:47:56Z

We have just released a beta release which should solve this issue.

@burbma if time permits, could you please verify whether this release solves your issue? Thanks.

(I will leave this issue open for the time being until we verify.)

scottbale added bug Something isn't working credentials labels Oct 12, 2023

scottbale added the enhancement New feature or request label Oct 12, 2023

gws mentioned this issue Oct 16, 2024

Support EC2 instance metadata service v2 (IMDSv2) #165

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IMDSv2 not supported #243

IMDSv2 not supported #243

burbma commented Oct 10, 2023 •

edited

Loading

scottbale commented Oct 12, 2023

tjg commented Nov 5, 2023 •

edited

Loading

brandonstubbs commented Jan 12, 2024

tlonist-sang commented Mar 6, 2024

dharrigan commented Mar 11, 2024

zerg000000 commented Mar 11, 2024

scottbale commented Oct 16, 2024

IMDSv2 not supported #243

IMDSv2 not supported #243

Comments

burbma commented Oct 10, 2023 • edited Loading

Dependencies

Description with failing test case

scottbale commented Oct 12, 2023

tjg commented Nov 5, 2023 • edited Loading

brandonstubbs commented Jan 12, 2024

tlonist-sang commented Mar 6, 2024

dharrigan commented Mar 11, 2024

zerg000000 commented Mar 11, 2024

scottbale commented Oct 16, 2024

burbma commented Oct 10, 2023 •

edited

Loading

tjg commented Nov 5, 2023 •

edited

Loading