Fix AWS IAM Roles for Service Accounts permission problem.

[serialx]

Amazon EKS supports IAM Roles for Service Accounts. It mounts tokens files to /var/run/secrets/eks.amazonaws.com/serviceaccount/token. Unfortunately, external-dns runs as 'nobody' so it cannot access this file. External DNS is then unable to make any AWS API calls to work:

time="2019-09-11T07:31:53Z" level=error msg="WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token\ncaused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied"

See: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html

Below are the file permissions mounted on External DNS pod:

~ $ ls -al /var/run/secrets/eks.amazonaws.com/serviceaccount/
total 0
drwxrwxrwt    3 root     root           100 Sep 11 06:40 .
drwxr-xr-x    3 root     root            28 Sep 11 06:40 ..
drwxr-xr-x    2 root     root            60 Sep 11 06:40 ..2019_09_11_06_40_49.865776187
lrwxrwxrwx    1 root     root            31 Sep 11 06:40 ..data -> ..2019_09_11_06_40_49.865776187
lrwxrwxrwx    1 root     root            12 Sep 11 06:40 token -> ..data/token
~ $ ls -al /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token
-rw-------    1 root     root          1028 Sep 11 06:40 /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token

Pinging @taharah for review.

[k8s-ci-robot]

Welcome @serialx!

It looks like this is your first PR to kubernetes-incubator/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-incubator/external-dns has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
• Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[serialx]

Updated CLA account settings.

[mikkeloscar]

Interesting. Do you know how the file permissions are for the Kubernetes service account token? Because this is mounted in a similar way and works with the nobody user.

[serialx]

Kubernetes tokens seems to be o+r

/run/secrets/kubernetes.io/serviceaccount $ ls -al
total 0
drwxrwxrwt    3 root     root           140 Sep 11 07:31 .
drwxr-xr-x    3 root     root            28 Sep 11 07:31 ..
drwxr-xr-x    2 root     root           100 Sep 11 07:31 ..2019_09_11_07_31_49.116680770
lrwxrwxrwx    1 root     root            31 Sep 11 07:31 ..data -> ..2019_09_11_07_31_49.116680770
lrwxrwxrwx    1 root     root            13 Sep 11 07:31 ca.crt -> ..data/ca.crt
lrwxrwxrwx    1 root     root            16 Sep 11 07:31 namespace -> ..data/namespace
lrwxrwxrwx    1 root     root            12 Sep 11 07:31 token -> ..data/token
/run/secrets/kubernetes.io/serviceaccount $ ls -al ..data/
total 12
drwxr-xr-x    2 root     root           100 Sep 11 07:31 .
drwxrwxrwt    3 root     root           140 Sep 11 07:31 ..
-rw-r--r--    1 root     root          1025 Sep 11 07:31 ca.crt
-rw-r--r--    1 root     root            12 Sep 11 07:31 namespace
-rw-r--r--    1 root     root           875 Sep 11 07:31 token

[mikkeloscar]

Very interesting, this means containers must run as root to use this feature. I don't think this is intended and we should make at least AWS aware.

[serialx]

Comparing the two codes doesn't show any hints on why they differ in permissions:

• https://github.com/aws/amazon-eks-pod-identity-webhook/blob/2fd4becd434f92edab01cecee30ea4b12af93ab7/pkg/handler/handler.go#L157-L177
• https://github.com/kubernetes/kubernetes/blob/bad4d0ee963a71994636ebaaca52dab54820abe6/plugin/pkg/admission/serviceaccount/admission.go#L512-L555

Strange.

[serialx]

Reported to the amazon-eks-pod-identity-webhook repository.

[alfredkrohmer]

The problem is here:
https://github.com/kubernetes/kubernetes/blob/cf76868b3430be015cc1e8443a216bf863a9b9f7/pkg/volume/projected/projected.go#L357

It's hardcoded to octal 0600. 🤦‍♂️

[serialx]

Damn. But both code uses ServiceAccountToken. Maybe the code path is different?

[alfredkrohmer]

No, this one is for projected service account tokens.

[alfredkrohmer]

See kubernetes/kubernetes#82573

[linki]

I was able to read a projected service account token with nobody by using securityContext.fsGroup:

      ...
      securityContext:
        fsGroup: 65534
      volumes:
      - name: token-vol
        projected:
          sources:
          - serviceAccountToken:
              path: token

$ whoami
nobody
$ ls -la /var/run/secrets/kubernetes.io/serviceaccount/..data/
-rw-r-----    1 root     nobody        1102 Sep 11 15:33 token

[marcincuber]

Any chances to get a timeline when this is going to be released?

[Raffo]

I'd defer to @linki , but I think that if a change of configuration will allow to do that, I'd rather change the docs to mention this than change the dockerfile.

[jqmichael]

65534

https://stackoverflow.com/questions/34831861/can-i-assume-that-nobody-is-65534

You can't. nobody has had at least a few different IDs across distros and time:

Historically, the user “nobody” was assigned UID -2 by several operating systems, although other values such as 2^(15)−1 = 32,767 are also in use, such as by OpenBSD. For compatibility between 16-bit and 32-bit UIDs, many Linux distributions now set it to be 2^(16)−2 = 65,534; the Linux kernel defaults to returning this value when a 32-bit UID does not fit into the return value of the 16-bit system calls. An alternative convention assigns the last UID of the range statically allocated for system use (0-99) to nobody: 99.

[serialx]

Yeah, but the docker file already specifies 65534, so it doesn't really matter does it?

https://github.com/kubernetes-incubator/external-dns/blob/master/Dockerfile#L36

[serialx]

Can confirm workaround by using securityContext. I've changed the PR to update AWS tutorial instead of changing the Dockerfile. Should work fine with future k8s updates too.

[linki]

/lgtm
/approve