Controller cannot find aws config Missing credentials in config

[wojtacha]

I've installed controller with:
helm install hello-name --set env.AWS_REGION=eu-west-1 --set securityContext.fsGroup=65534 --set serviceAccount.annotations."eks.amazonaws.com/role-arn"='arn:aws:iam::AWS-ACCOUNT:role/ROLE-NAME' external-secrets/kubernetes-external-secrets

kubeclt apply this file
secret.yaml:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: gtp-secrets
spec:
  backendType: systemManager
  roleArn: arn:aws:iam::AWS-ACCOUNT:role/ROLE-NAME
  data:
    - key: /hello-service/password
      name: password
  template:
    metadata:
      annotations:
        cat: cheese
      labels:
        dog: farfel

when I kubectl describe pod

AWS_ROLE_ARN: arn:aws:iam::AWS-ACCOUNT:role/ROLE-NAME
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

still got this issue:

{"level":30,"time":1582206597344,"pid":17,"hostname":"gtp-secret-controller-kubernetes-external-secrets-5f788c485l9vx","msg":"fetching` secret property /hello-service/password with role: arn:aws:iam::AWS-ACCOUNT:role/ROLE-NAME","v":1}
{"level":50,"time":1582206597856,"pid":17,"hostname":"gtp-secret-controller-kubernetes-external-secrets-5f788c485l9vx","message":"Missing credentials in config","statusCode":502,"retryable":true,"time":"2020-02-20T13:49:57.856Z","code":"CredentialsError","originalError":{"message":"Could not load credentials from any providers","statusCode":502,"retryable":true,"time":"2020-02-20T13:49:57.856Z","code":"CredentialsError","originalError":{"message":"EC2 Metadata token request returned error","statusCode":502,"retryable":true,"time":"2020-02-20T13:49:57.856Z"}},"msg":"failure while polling the secret hello-namespace/hello-secrets","stack":"CredentialsError: Missing credentials in config\n    at IncomingMessage.<anonymous> (/app/node_modules/aws-sdk/lib/util.js:895:34)\n    at IncomingMessage.emit (events.js:215:7)\n    at IncomingMessage.EventEmitter.emit (domain.js:476:20)\n    at endReadableNT (_stream_readable.js:1183:12)\n    at processTicksAndRejections (internal/process/task_queues.js:80:21)","type":"Error","v":1}

[Flydiverny]

I believe the roleArn: arn:aws:iam:::role/ in your secret might be the issue. I believe we currently don't have a working implementation for this when using Iam roles for service account as the aws sdk require other assume role calls with the Web identity.

Edit for future readers, my assumption here was wrong.

[wojtacha]

I forgot to mention that each node in k8s (that is essentially ec2) has policy that allows to read ssm and aws secrets.
Which is first point of AWS based backends in the readme

"1. Granting your nodes explicit access to your secrets using the node instance role (easy for experimentation, not recommended)"

I dont know if that changes anything.

Cheers

[ricardo-mela-ms]

I disabled instance profile IAM roles for the pods https://docs.aws.amazon.com/eks/latest/userguide/restrict-ec2-credential-access.html and am using IAM roles for service accounts, but the controller insists on using the instance profile.

"message":"Missing credentials in config","errno":"ETIMEDOUT","code":"CredentialsError","syscall":"connect","address":"169.254.169.254","port":80,"time":"2020-05-26T09:08:29.749Z","originalError":{"message":"Could not load credentials from any providers","errno":"ETIMEDOUT","code":"CredentialsError","syscall":"connect","address":"169.254.169.254","port":80

I can see the credentials being there as set | grep AWS shows me the role and the token.

The secret no longer has the roleArn set but it still does not work.

Could it be that IAM roles for service account does not work at all?

[ricardo-mela-ms]

In my case, I tested earlier by allowing the instance profile to assume the role mentioned in roleArn and it worked just fine.

[dalgibbard]

Is it possible this is related to #254 for Fargate/WebIdentity support? Looks like similar errors relating to not being able to reach the EC2 metadata API (which isn't available on Fargate instances).

If so; this fix is merged to master now, waiting on a release :)

[wilkej]

Hey everyone,
I'm also trying to get external secrets running in a EKS Fargate cluster. Additionally I like to mention that my vpc has no internet connection but VPC endpoints for ec2, secretsmanager, sts.

I'm confused if the issue is solved and have the impression it isn't solved.

As far as I understand in #254 it was mentioned that it is solved, which relates to the comment from the first of July. In #442 it looks like #254 was reverted.

I deployed external secret with the docker image godaddy/kubernetes-external-secrets:5.2.0

Still the issue is for me:

"message":"Missing credentials in config","errno":"ETIMEDOUT","code":"CredentialsError","syscall":"connect","address":"169.254.169.254","port":80,"time":"...","originalError":{"message":"Could not load credentials from any providers","errno":"ETIMEDOUT","code":"CredentialsError","syscall":"connect","address":"169.254.169.254","port":80,"time":"...","originalError":{"message":"Missing credentials in config","errno":"ETIMEDOUT","code":"CredentialsError","syscall":"connect","address":"169.254.169.254","port":80,"time":"...","originalError":{"message":"Could not load credentials from any providers","errno":"ETIMEDOUT","code":"CredentialsError","syscall":"connect","address":"169.254.169.254","port":80,"time":"...","originalError":{"message":"EC2 Metadata roleName request returned error","errno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect","address":"169.254.169.254","port":80,"time":"2...","originalError":{"errno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect","address":"169.254.169.254","port":80,"message":"connect ETIMEDOUT 169.254.169.254:80"}}}}},"stack":"Error: connect ETIMEDOUT 169.254.169.254:80\n at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1141:16)","type":"Error","msg":"failure while polling the secret ..."}

Also the environment variables are set fine (I think)

• EXTERNAL_SECRETS_KUBERNETES_EXTERNAL_SECRETS_SERVICE_HOST=172.20.173.101
• EXTERNAL_SECRETS_KUBERNETES_EXTERNAL_SECRETS_PORT_3001_TCP=tcp://172.20.173.101:3001
• KUBERNETES_SERVICE_PORT=443
• KUBERNETES_PORT=tcp://172.20.0.1:443
• LOG_LEVEL=info
• NODE_VERSION=12.18.2
• LOG_MESSAGE_KEY=msg
• METRICS_PORT=3001
• KUBE_DNS_SERVICE_PORT_DNS_TCP=53
• HOSTNAME=
• YARN_VERSION=1.22.4
• VAULT_ADDR=http://127.0.0.1:8200
• SHLVL=2
• HOME=/home/node
• AWS_ROLE_ARN=
• EXTERNAL_SECRETS_KUBERNETES_EXTERNAL_SECRETS_SERVICE_PORT=3001
• EXTERNAL_SECRETS_KUBERNETES_EXTERNAL_SECRETS_PORT=tcp://172.20.173.101:3001
• AWS_WEB_IDENTITY_TOKEN_FILE=
• KUBE_DNS_SERVICE_HOST=172.20.0.10
• POLLER_INTERVAL_MILLISECONDS=60000
• AWS_DEFAULT_REGION=eu-central-1
• KUBE_DNS_PORT=udp://172.20.0.10:53
• KUBE_DNS_SERVICE_PORT=53
• TERM=xterm
• KUBERNETES_PORT_443_TCP_ADDR=172.20.0.1
• PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
• NPM_CONFIG_LOGLEVEL=info
• KUBE_DNS_PORT_53_TCP_ADDR=172.20.0.10
• KUBERNETES_PORT_443_TCP_PORT=443
• KUBERNETES_PORT_443_TCP_PROTO=tcp
• KUBE_DNS_PORT_53_UDP_ADDR=172.20.0.10
• EXTERNAL_SECRETS_KUBERNETES_EXTERNAL_SECRETS_SERVICE_PORT_PROMETHEUS=3001
• KUBE_DNS_PORT_53_TCP_PORT=53
• KUBE_DNS_PORT_53_TCP_PROTO=tcp
• KUBE_DNS_PORT_53_UDP_PORT=53
• EXTERNAL_SECRETS_KUBERNETES_EXTERNAL_SECRETS_PORT_3001_TCP_ADDR=172.20.173.101
• AWS_REGION=eu-central-1
• KUBE_DNS_PORT_53_UDP_PROTO=udp
• KUBE_DNS_SERVICE_PORT_DNS=53
• EXTERNAL_SECRETS_KUBERNETES_EXTERNAL_SECRETS_PORT_3001_TCP_PORT=3001
• KUBERNETES_PORT_443_TCP=tcp://172.20.0.1:443
• KUBERNETES_SERVICE_PORT_HTTPS=443
• EXTERNAL_SECRETS_KUBERNETES_EXTERNAL_SECRETS_PORT_3001_TCP_PROTO=tcp
• KUBERNETES_SERVICE_HOST=172.20.0.1
• PWD=/app
• KUBE_DNS_PORT_53_TCP=tcp://172.20.0.10:53
• KUBE_DNS_PORT_53_UDP=udp://172.20.0.10:53
• NODE_ENV=production

Any ideas?

[Flydiverny]

Its correct it was reverted as it broke other things and wasn't needed. Make sure security context is properly set.
As written here #442 (comment) it should all work with the right configuration :)

[wilkej]

I saws your comment and my use case is slightly different. In you example you assume a role to access the external secret. In my case the IRSA role is allowed to directly access the external secret. This was working fine in a EKS with EC2 instances.

The documentation says: "Additionally, you can specify a roleArn which will be assumed before retrieving the secret." I understand it isn't a requirement. The security context is right. Here is my helm chart yaml.

customResourceManagerDisabled: false

crds:
  create: false

env:
  AWS_REGION: eu-central-1
  AWS_DEFAULT_REGION: eu-central-1
  POLLER_INTERVAL_MILLISECONDS: 60000 
  LOG_LEVEL: info
  LOG_MESSAGE_KEY: 'msg'
  METRICS_PORT: 3001
  VAULT_ADDR: http://127.0.0.1:8200


rbac:
  create: true

serviceAccount:
  create: true
  annotations: {
    eks.amazonaws.com/role-arn: arn:aws:iam::REMOVED:role/REMOVED
  }
  name: REMOVED

replicaCount: 1

image:
  repository: REMOVED.dkr.ecr.eu-central-1.amazonaws.com/godaddy/kubernetes-external-secrets
  tag: 5.2.0
  pullPolicy: IfNotPresent
  imagePullSecrets: []

nameOverride: ""
fullnameOverride: ""

podAnnotations: {}
podLabels: {}

dnsConfig: {}

securityContext:
  runAsNonRoot: false
  fsGroup: 65534

resources: {}

nodeSelector: {}

tolerations: []

affinity: {}

serviceMonitor:
  enabled: false
  interval: "30s"
  namespace:

I'll try later to see if the assume role for a secret may change the behavior.

[dalgibbard]

I still vote for a flag to allow explicitly using WebIdentity 🤷‍♂️ :) maybe a try/catch to use web identity if assume role fails and the web identity file is present?

[wilkej]

I think my issue lies somewhere else. In the AWS console I get the message, that my Role for the pod was Not accessed in the tracking period.

I installed the aws cli in the pod and get the following error:

aws sts get-caller-identity

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

I need to find out why this isn't working and hopefully the issue will then solve itself.

[wilkej]

I found the issue. In my trust relationship of the role I had the condition key

oidc.eks.eu-central-1.amazonaws.com/id/...:aud configured but it needs to be

oidc.eks.eu-central-1.amazonaws.com/id/...:sub

Thank you for your efforts. From my point of view it looks like the flag allow explicitly using WebIdentity isn't required, as it is working out of the box

[dalgibbard]

Ok cool, I'll go back in my box 😄