[EKS] [BAD-DECISION]: EKS Pod Identity agent daemonset mapped to node-port 80

[ChrisMcKee]

Copied over from aws/containers-roadmap#2356

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
What do you want us to build?

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

We tried to install the eks-pod-identity-agent addon so that we could set the auth config to allow both options.
The addon installs as a daemonset with HostNetwork set to true, pod permissions to map to the node, and a default port set to 80.
The instant that the service started to install, all of our Haproxy ingress pods were evicted so that the identity service could map to port 80.

I'd love to know the rationale that went into choosing to map the node-port to what is literally the main http port; and then not to document how to change it to avoid collisions. Through all the documentation that mentions it the only warning is here https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html#pod-id-considerations and it's a note rather than informative. The majority of links go straight to https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html which doesnt mention it at all.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: eks-pod-identity-agent
  namespace: kube-system
spec:
  template:    
    spec:
      containers:
        - name: eks-pod-identity-agent
          image: >-
            602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/eks-pod-identity-agent:0.1.6
          command:
            - /go-runner
            - /eks-pod-identity-agent
            - server
          args:
            - '--port'
            - '80'
            - '--cluster-name'
            - prod-eks-cluster
            - '--probe-port'
            - '2703'
          ports:
            - name: proxy
              containerPort: 80
              protocol: TCP
            - name: probes-port
              containerPort: 2703
              protocol: TCP
          securityContext:
            capabilities:
              add:
                - CAP_NET_BIND_SERVICE
      hostNetwork: true

Are you currently working around this issue?
How are you currently solving this problem?

Uninstalled the Addon

[ChrisMcKee]

[tanvp112]

This design will also break Fargate only cluster: terraform-aws-modules/terraform-aws-eks#2850 (comment)

[manu-sucerquia-payu-gpo]

This also breaks when enabling the addon in a cluster running Project Contour's Envoy proxy (they use the same ports in hostNetwork). Has anyone found a work around this issue or is there any news?

[suhaspotluri]

Also breaking our Traefik daemonset installation as we install Traefik with hostPort 80 and 443

[joncolby]

does eks-pod-identity-agent really require listening on port 80 ? what is exactly this port used for? if another port were configured, will everything still work?

[taraspos]

does eks-pod-identity-agent really require listening on port 80 ? what is exactly this port used for? if another port were configured, will everything still work?

https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html#pod-id-considerations

The EKS Pod Identity Agent uses the hostNetwork of the node and it uses port 80 and port 2703 on a link-local address on the node. This address is 169.254.170.23 for IPv4 and [fd00:ec2::23] for IPv6 clusters.

If you disable IPv6 addresses, or otherwise prevent localhost IPv6 IP addresses, the agent can't start. To start the agent on nodes that can't use IPv6, follow the steps in Disable IPv6 in the EKS Pod Identity Agent to disable the IPv6 configuration.

That's the port that actually being hit when pods requesting credentials via GET http://169.254.170.23:80

[jvdadda]

does eks-pod-identity-agent really require listening on port 80 ? what is exactly this port used for? if another port were configured, will everything still work?

With current config, yes, they can change it to target a different port but it seems not important enough to work on it yet. 🙄

[cspigner]

+1 request to be able to change the listening port at deployment for pod-identity-agent ds.

This directly conflicts with being able to configure ingress-controller pod to listen on port 80 on the same node and causes extremely volatile behavior with autoscalers like karpenter.

[MDBeudekerCN]

I had this same behavior, the weird thing is that for my cluster I get only 1 unstable pod-identity-agent, while the rest of the pods in the daemonset deploys just fine.

There is also an nginx-ingress daemonset running, but I'm confused if the ingress controller blocks the port if 2 of the 3 pods schedule just fine.

When I look at all the hostPorts in the entire cluster I see that only the agents have a explicit port 80 assigned.

I must be missing something crucial, but I am very curious why it does work for some of the pods 🤔

[MDBeudekerCN]

Update, so apparently in this setup a certain crucial pod was assigned to NodePort 80, so only one node could not have the PodIdentity enabled. Not sure if and how I can work around it but time pressure wise we cannot work forward with this

[changhyuni]

Specifying port 80 as a hostPort seems to be incorrect.
I can't use the already occupied port 80 either.
Why use port 80 as a hostPort? Intra-node routing is handled by pod-id-link0 anyway.

[shayro12]

I've found a simple workaround for this situation:

1. Change the port on the DaemonSet from 80 to another port, for example, 8088.

2. Add the following environment variables to your pods:

   - name: AWS_CONTAINER_CREDENTIALS_FULL_URI
     value: "http://169.254.170.23:8088/v1/credentials"
   - name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
     value: "/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token"

[gohmc]

The managed addon doesn't support changing the port. For self managed, currently "80" is hard coded, example the helm chart. Port # change is only the first half, the second half is the pod identity webhook must also inject the correct value.