Unable to Obtain IAM Service Credentials from OpenID/EKS

[ijaveed]

Describe the bug

We are using EKS & are now moving away from using worker node instance profiles to provide credentials to using IAM service roles for pods.

I have followed all the instructions in relation to IAM service roles for Pods -->
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html

We have tested this with the latest version with awscli which is able to obtain the credentials via oidc provider..

But when we use the aws-sdk-js, it does not even attempt to connect to the oidc provider , but rather still wants to try & connect to the instance metadata service at 169.254.... ( as per the docs , the worker node has an iptables rules which blocks access to it from all eni's)

the correct variables are present & the uid within the container is able to read the credentials file...

AWS_ROLE_ARN=arn:aws:iam::1234567890:role/demo_payment_service_role_001
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_REGION=eu-west-2


aws-sdk-js -->  Version: 2.610.0

Receive Error { Error: connect ETIMEDOUT 169.254.169.254:80
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1097:14)
  message: 'Missing credentials in config',
  errno: 'ETIMEDOUT',
  code: 'CredentialsError',
  syscall: 'connect',
  address: '169.254.169.254',
  port: 80,
  time: 2020-02-08T09:33:47.079Z,
  originalError:
   { message: 'Could not load credentials from any providers',
     errno: 'ETIMEDOUT',
     code: 'CredentialsError',
     syscall: 'connect',
     address: '169.254.169.254',
     port: 80,
     time: 2020-02-08T09:33:47.079Z,
     originalError:
      { message: 'EC2 Metadata roleName request returned error',
        errno: 'ETIMEDOUT',
        code: 'ETIMEDOUT',
        syscall: 'connect',
        address: '169.254.169.254',
        port: 80,
        time: 2020-02-08T09:33:47.079Z,
        originalError: [Object] } 
       } 
    }

[ajredniwja]

Hey @ijaveed can you please share the code that you are using to reproduce this?

[johnwatson484]

I have been working with @ijaveed on this issue. Here is the code we have.

Within our Helm chart we have defined the following service.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxx:role/xxx

We have reference to this in the deployment.

template:
    spec:
      serviceAccountName: my-service-account

I am aware of the issue of running this as non-root and the work around using the fsGroup, however for the sake of experimentation I have tried this as running as root and non-root.

I can see in the pod that the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE have both been correctly set. I have also added the AWS_REGION environment variable to see if that made any difference.

AWS_REGION:  eu-west-2
AWS_ROLE_ARN: arn:aws:iam::xxx:role/xxx
AWS_WEB_IDENTITY_TOKEN_FILE:  path/to/token

For the purposes of trying to get this to work, we have a simplified version of our queue consuming logic. It's essentially the code snippets taken from AWS documentation.

var sqs = new AWS.SQS()

    var params = {
      AttributeNames: [
        'SentTimestamp'
      ],
      MaxNumberOfMessages: 10,
      MessageAttributeNames: [
        'All'
      ],
      QueueUrl: queueURL,
      VisibilityTimeout: 20,
      WaitTimeSeconds: 0
    }

    sqs.receiveMessage(params, function (err, data) {
      if (err) {
        console.log('Receive Error', err)
      } else if (data.Messages) {
        var deleteParams = {
          QueueUrl: queueURL,
          ReceiptHandle: data.Messages[0].ReceiptHandle
        }
        sqs.deleteMessage(deleteParams, function (err, data) {
          if (err) {
            console.log('Delete Error', err)
          } else {
            console.log('Message Deleted', data)
          }
        })
      }
    })

We have also tried supplying dummy access keys as advised elsewhere with no luck.

var sqs = new AWS.SQS({
      accessKeyId: dummyKeyId,
      secretAccessKey: dummyKey
    })

[aRobinson-R7]

@ijaveed @johnwatson484 I have been having this issue myself when using the S3 client.
The way I have been able to make it work is by explicitly setting the credentialProvider used by the auth client to the defaultCredentialProviderChain

e.g config.credentialProvider = new Aws.CredentialProviderChain()

Docs

[johnwatson484]

@aRobinson-R7 thanks, we had tried that with no luck.

We raised a support ticket with AWS who supplied a code sample that should work. It was pretty similar to what I'd tried above.

However, without making any changes it starting magically working so I'm not sure if they also made changes to the eu-west-2 region to support this.

Either way it confirmed this is not an SDK issue.

[ajredniwja]

However, without making any changes it starting magically working so I'm not sure if they also made changes to the eu-west-2 region to support this.

Either way it confirmed this is not an SDK issue.

Closing this issue now, please reach out for additional questions.

[danm-pb]

I am having the same issue with node and IAM roles - I've confirmed the serviceAccount and IAM role permissions are correct by testing in an amaxonlinux container with aws-cli.
Could @johnwatson484 expand on what code snippet AWS suggested pls?

[johnwatson484]

I am having the same issue with node and IAM roles - I've confirmed the serviceAccount and IAM role permissions are correct by testing in an amaxonlinux container with aws-cli.
Could @johnwatson484 expand on what code snippet AWS suggested pls?

Sure, this was the response we got back.

Open an interactive session with the pod in question and enter the following commands.

--node -> It will take us to a terminal with ">"
-- in order to check the CallerIdentity, we have installed aws-sdk and checked the CallerIdentity by running each line in turn:

const aws = require('aws-sdk');
const sts = new aws.STS({region: "us-east-1"});
sts.getCallerIdentity({}, function (error,data){console.log(data)});

It should return something along the lines of the below and you should see the response object at the bottom returning the IAM account details.

/usr/src/app $ node
> 
Error: Could not open history file.
REPL session history will not be persisted.
> const aws = require('aws-sdk');
undefined
> const sts = new aws.STS({region: "eu-west-2"});
undefined
> sts.getCallerIdentity({}, function (error,data){console.log(data)});
Request {
  domain:
   Domain {
     domain: null,
     _events:
      [Object: null prototype] {
        removeListener: [Function: updateExceptionCapture],
        newListener: [Function: updateExceptionCapture],
        error: [Function: debugDomainError] },
     _eventsCount: 3,
     _maxListeners: undefined,
     members: [] },
  service:
   Service {
     config:
      Config {
        credentials: null,
        credentialProvider: [CredentialProviderChain],
        region: 'eu-west-2',
        logger: null,
        apiVersions: {},
        apiVersion: null,
        endpoint: 'https://sts.amazonaws.com',
        httpOptions: [Object],
        maxRetries: undefined,
        maxRedirects: 10,
        paramValidation: true,
        sslEnabled: true,
        s3ForcePathStyle: false,
        s3BucketEndpoint: false,
        s3DisableBodySigning: true,
        s3UsEast1RegionalEndpoint: 'legacy',
        s3UseArnRegion: undefined,
        computeChecksums: true,
        convertResponseTypes: true,
        correctClockSkew: false,
        customUserAgent: null,
        dynamoDbCrc32: true,
        systemClockOffset: 0,
        signatureVersion: 'v4',
        signatureCache: true,
        retryDelayOptions: {},
        useAccelerateEndpoint: false,
        clientSideMonitoring: false,
        endpointDiscoveryEnabled: false,
        endpointCacheSize: 1000,
        hostPrefixEnabled: true,
        stsRegionalEndpoints: undefined },
     isGlobalEndpoint: true,
     endpoint:
      Endpoint {
        protocol: 'https:',
        host: 'sts.amazonaws.com',
        port: 443,
        hostname: 'sts.amazonaws.com',
        pathname: '/',
        path: '/',
        href: 'https://sts.amazonaws.com/' },
     _events: { apiCallAttempt: [Array], apiCall: [Array] },
     MONITOR_EVENTS_BUBBLE: [Function: EVENTS_BUBBLE],
     CALL_EVENTS_BUBBLE: [Function: CALL_EVENTS_BUBBLE],
     _clientId: 1 },
  operation: 'getCallerIdentity',
  params: {},
  httpRequest:
   HttpRequest {
     method: 'POST',
     path: '/',
     headers:
      { 'User-Agent': 'aws-sdk-nodejs/2.610.0 linux/v10.15.3 callback' },
     body: '',
     endpoint:
      Endpoint {
        protocol: 'https:',
        host: 'sts.amazonaws.com',
        port: 443,
        hostname: 'sts.amazonaws.com',
        pathname: '/',
        path: '/',
        href: 'https://sts.amazonaws.com/',
        constructor: [Function] },
     region: 'us-east-1',
     _userAgent: 'aws-sdk-nodejs/2.610.0 linux/v10.15.3 callback' },
  startTime: 2020-02-19T10:42:53.391Z,
  response:
   Response {
     request: [Circular],
     data: null,
     error: null,
     retryCount: 0,
     redirectCount: 0,
     httpResponse:
      HttpResponse {
        statusCode: undefined,
        headers: {},
        body: undefined,
        streaming: false,
        stream: null },
     maxRetries: 3,
     maxRedirects: 10 },
  _asm:
   AcceptorStateMachine {
     currentState: 'validate',
     states:
      { validate: [Object],
        build: [Object],
        afterBuild: [Object],
        sign: [Object],
        retry: [Object],
        afterRetry: [Object],
        send: [Object],
        validateResponse: [Object],
        extractError: [Object],
        extractData: [Object],
        restart: [Object],
        success: [Object],
        error: [Object],
        complete: [Object] } },
  _haltHandlersOnError: false,
  _events:
   { validate:
      [ [Function],
        [Function: optInRegionalEndpoint],
        [Function],
        [Function: VALIDATE_REGION],
        [Function: BUILD_IDEMPOTENCY_TOKENS],
        [Function: VALIDATE_PARAMETERS] ],
     afterBuild:
      [ [Function],
        [Function: SET_CONTENT_LENGTH],
        [Function: SET_HTTP_HOST] ],
     restart: [ [Function: RESTART] ],
     sign: [ [Function], [Function], [Function] ],
     validateResponse: [ [Function: VALIDATE_RESPONSE], [Function] ],
     send: [ [Function] ],
     httpHeaders: [ [Function: HTTP_HEADERS] ],
     httpData: [ [Function: HTTP_DATA] ],
     httpDone: [ [Function: HTTP_DONE] ],
     retry:
      [ [Function: FINALIZE_ERROR],
        [Function: INVALIDATE_CREDENTIALS],
        [Function: EXPIRED_SIGNATURE],
        [Function: CLOCK_SKEWED],
        [Function: REDIRECT],
        [Function: RETRY_CHECK],
        [Function: API_CALL_ATTEMPT_RETRY] ],
     afterRetry: [ [Function] ],
     build: [ [Function: buildRequest] ],
     extractData: [ [Function: extractData], [Function: extractRequestId] ],
     extractError: [ [Function: extractError], [Function: extractRequestId] ],
     httpError: [ [Function: ENOTFOUND_ERROR] ],
     success: [ [Function: API_CALL_ATTEMPT] ],
     complete: [ [Function: API_CALL], [Function] ] },
  emit: [Function: emit],
  API_CALL_ATTEMPT: [Function: API_CALL_ATTEMPT],
  API_CALL_ATTEMPT_RETRY: [Function: API_CALL_ATTEMPT_RETRY],
  API_CALL: [Function: API_CALL] }
> { ResponseMetadata: { RequestId: 'db0a2ed4-5970-4c7f-96cf-07e29b3c7196' },
  UserId: 'USERID:token-file-web-identity',
  Account: 'ACCOUNT',
  Arn:
   'arn:aws:sts::ACCOUNT:assumed-role/ROLE/token-file-web-identity' }

If it returns an assumed identity then the issue is in the application code's use of the SDK. If it doesn't then something is incorrect somewhere in the OIDC, IAM, trust policy area.

It's also important to remember the workaround using fsGroup in the security context of the pod if you're running your container as a non root user.

For this to work you must have this set.

Below is a trimmed down version of our Helm chart we use for deployment with the value hard coded in for an example. If you're not using Helm then the equivalent Kubernetes pod yaml will be similar.

apiVersion: extensions/v1beta1
kind: Deployment
  template:    
      securityContext:
        fsGroup: 1000

[danm-pb]

Many thx @johnwatson484 much appreciated. Gives me something to try during lockdown! :)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.