[EKS] [security]: Use IRSA to provide privileges to VPC CNI DaemonSet instead of EC2 Instance Role

[otterley]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Please note that I am not reporting an immediate exploitable vulnerability, but rather, a change request to implement a best practice and reduce permission scope to that of least privilege.

Currently, EC2 Instances that comprise EKS managed nodegroups are expected to have an IAM Policy (AmazonEKS_CNI_Policy) assigned to them via their EC2 Instance Role that allows them broad access to manage Elastic Network Interfaces (ENIs). The requirement is discussed in our documentation.

This Instance Role and associated Policy were once required to allow the CNI controller pods, which are deployed as a DaemonSet on each worker node, to manage ENI associations with the Instances comprising the worker nodes. However, supplying each Instance these permissions implicitly provided the same permissions to all containers running on the node, whether those containers actually needed them or not. (While this privilege could be restricted from containers by denying them access to the EC2 Metadata Endpoint, this action would also deny them access to other information or credentials they might legitimately require.)

With the availability of IAM Roles for Service Accounts (IRSA) this requirement is no longer needed. Instead, the CNI DaemonSet pods should be associated with a ServiceAccount that is tied to an IAM Role having permission to manipulate the ENIs.

Once that is done, the worker node role will no longer need the ability to manage ENIs, thereby improving the overall security posture for all customers.

[mikestef9]

We do provide instructions on our docs on how to accomplish this task. We'll investigate enabling this by default

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-cni-walkthrough.html

Another note, next gen CNI #398 will remove the need for the long running daemon on every node that requires broad VPC permissions

[3oris]

@mikestef9 ,

following the steps in https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-cni-walkthrough.html is cumbersome, because it requires us to create a new role for every worker/node group to be started. So I need to attach an unnecessary (see below) policy, just to remove it right after startup of the worker nodes. Moreover this doesn't work well with tools like terraform.

This is how the AmazonEKS_CNI_Policy becomes unnecessary on the worker nodes:

1. Create an EKS control plane
2. Finalize OIDC Provider Setup
3. Create a role named "eks-cni-plugin" with the AmazonEKS_CNI_Policy, OIDC principal and condition attached to it,
4. Annotate the "aws-node" service account in the kube-system namespace with the "eks-cni-plugin" role
5. Create a (managed) node group

Once the node group is created, do a:
kubectl describe pods -n kube-system aws-node-xyzabc | grep AWS_ROLE_ARN

It will show:
AWS_ROLE_ARN: arn:aws:iam::XXX:role/eks-cni-plugin

As you can see, the AmazonEKS_CNI_Policy is not even needed in the worker node role. Moreover, considering the steps above we think that AmazonEKS_CNI_Policy is displaced on the worker nodes, since conceptually it belongs to the "aws-node" daemonset which is by design part of the control plane setup.

So essentially, step 1 to 4 could be part of the EKS managed service control plane setup, it just requires one more role arn/name as an input parameter (the role carrying the AmazonEKS_CNI_Policy)

There might also be an easy migration path for this: if the additional role arn/name is given as input provision the control plane like above (and remove the enforcement of the CNI policy on the node role), else keep enforcement for the CNI policy on the node role.

What do you think?

[brycecarman]

You are absolutely correct that AmazonEKS_CNI_Policy is not required on the worker node roles if you are using IAM Roles for service accounts on the CNI. By default, the CNI does not use IRSA so the policy is required to prevent the common case where the user uses a role that is missing the policy which would not allow the worker nodes to become ready.

There is also a bit of a chicken/egg problem. To create the IRSA role for the CNI, you need the IAM OIDC identity provider. To create that you need the cluster to have been created. It is a very priviledged operation to create the roles and OIDC identity provider so it is problematic for EKS to create them for the user in many cases.

We are evaluating on how to make the situation more streamlined.

[3oris]

@brycecarman -- That 5 steps plan (see above) was specifically meant to show what needs to be done in which order to resolve the chicken/egg situation, meaning how everything can be set up before the node groups come into existence.

[tabern]

This will be solved by #252

[inductor]

@tabern Hi. Has this been solved since you've got AWS CNI in add-on?

[Norfolc]

Hello,
I'm facing issues after removing the CNI policy from the node IAM role as described in https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-cni-walkthrough.html

We have a cluster with both Linux and Windows nodes.
To enable windows support we must install vpc-resource-controller as described in https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html

Removing AmazonEKS_CNI_Policy from the node role prevents vpc-resource-controller to assign private IP addresses to Windows nodes.
As a result windows pods are stuck in ContainerCreating state with errors like this:
Warning FailedCreatePodSandBox 16s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f5...3c" network for pod "windows-server-iis-865b9498b9-fvft7": networkPlugin cni failed to set up pod "windows-server-iis-865b9498b9-fvft7_default" network: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4Address

I tried to add IRSA to vpc-resource-controller but it did not help - it still uses node role.

[Norfolc]

Sorry, I was not exactly right: vpc-resource-controller does not uses node role when adding IRSA.
It just requires more permissions in addition to AmazonEKS_CNI_Policy: ec2:DescribeSubnets and ec2:DescribeRouteTables.

May be it requires more permission, but these are just the permissions that I've found in CloudTrail that has failed.

I would be fine if someone from AWS team could confirm and document what exact permission set is required for vpc-resource-controller to run correctly.

Also documentation about windows nodes is misleading about this topic: https://docs.aws.amazon.com/eks/latest/userguide/launch-windows-workers.html
4. (Optional) If the AmazonEKS_CNI_Policy managed IAM policy is attached to your Amazon EKS node IAM role, we recommend assigning it to an IAM role that you associate to the Kubernetes aws-node service account instead. For more information, see Configuring the Amazon VPC CNI plugin to use IAM roles for service accounts.

[ChrisMcKee]

Also documentation about windows nodes is misleading about this topic: https://docs.aws.amazon.com/eks/latest/userguide/launch-windows-workers.html 4. (Optional) If the AmazonEKS_CNI_Policy managed IAM policy is attached to your Amazon EKS node IAM role, we recommend assigning it to an IAM role that you associate to the Kubernetes aws-node service account instead. For more information, see Configuring the Amazon VPC CNI plugin to use IAM roles for service accounts.

I imagine these were missed but previously covered in the AmazonEKSWorkerNodePolicy; make sense if AWS duplicated that permission in the CNI policy imo.

[mikestef9]

Going to close this issue now that EKS Pod Identity has launched, which allows you to setup IAM permissions ahead of cluster launch time

https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html

[mithunsl]

@mikestef9

The AWS documentation states that the EKS Pod Identity cannot be used with VPC CNI.

You can't use EKS Pod Identities with:

Amazon EKS add-ons that need IAM credentials. The EKS add-ons can only use IAM roles for service accounts instead. The list of EKS add-ons that use IAM credentials include:

- Amazon VPC CNI plugin for Kubernetes

- AWS Load Balancer Controller

- The CSI storage drivers: EBS CSI, EFS CSI, Amazon FSx for Lustre CSI driver, Amazon FSx for NetApp ONTAP CSI driver, Amazon FSx for OpenZFS CSI driver, Amazon File Cache CSI driver

[joebowbeer]

@mithunsl see note that differentiates self-managed add-ons from EKS add-ons:

If these controllers, drivers, and plugins are installed as self-managed add-ons instead of EKS add-ons, they support EKS Pod Identities as long as they are updated to use the latest AWS SDKs.