[EKS] [request]: provide taints and labels for managed nodes from the start

[dekarl]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
provide taints and labels from the beginning of a nodes lifecycle, as if they had been on the kubelet command line

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We switched to providing taints and labels via the managed node group after this feature was added in #917.
It appears that taints and labels are only applied around half a minute after the node joins the cluster. This has multiple undesirable side effect.

• when adding capacity from different node groups for different workloads the scheduler places indifferent pods to nodes that later get taints instead of our catch-all node group.
• the late addition of labels breaks the display of pricing information in the eks-node-viewer for managed nodes, see new on-demand instances show the spot price (on managed 1.21 node groups) awslabs/eks-node-viewer#36

Are you currently working around this issue?
Not at all.
Possible workaround are:

• falling back to specifying labels and taints on the kubelet command line
• adding a descheduler to all clusters

Additional context
Due to the different way of Karpenter to bring up nodes its working as expected with it.
Tested with EKS managed node groups version 1.23 platform version eks.6

[mmerkes]

I assume that you're using a custom AMI with managed node groups? For standard or custom launch template nodegroups, we update the user data to set --register-with-taints and --node-labels in the kubelet extra args. However, for custom AMI nodegroups, that's not an option because we don't touch the userdata. As a result of that, we can't immediately taint/label nodes as part of the registration process. We have an asynchronous process that taints/labels them independent from registration.

One workaround would be to set those fields in your user data to ensure that they get set right way.

[hileef]

We were experiencing similar symptoms (node taints applied "late" by a lambda instead of on cluster join),
and as explained by AWS Support, and additionally pointed out just above, the --register-with-taints and --node-labels kubelet flags required to have the expected behaviour are meant to be injected via the launch template's user-data ;

This means that if for any reason you override the MNG's ASG's launch-template's user-data (that was our mistake due to legacy internal Infrastructure-as-Code), these flags need to be added to the user-data you provide in order to get the intended behaviour 🙂

[jeff-armis]

It feels misleading to provide a taints field in CreateNodeGroup API, but then also require the user to supply --register-with-taints in UserData. Wouldn't it be rather trivial for AWS to add a aws eks describe-nodegroup in the EKS bootstrap script to automatically handle the --register-with-taints instead?

[mmerkes]

@jeff-armis We've discussed adding what you describe to the bootstrap script. Besides the effort to implement that and test it, one limitation is that it won't work for customers that don't have connectivity to the EKS API within their nodes. That's not a huge deal and should still solve the issue for many customers, but it won't be a one size fits all fix and it will need to be behind a flag.

[jeff-armis]

Yes, it would have the same behavior as other supported flags like APISERVER_ENDPOINT and B64_CLUSTER_CA (a customer can elect to supply these, OR, if they have access to the EKS API, let the bootstrap script do it for them). I think labels should also be handled this way.
As a user, I can't think of a use case where it would be acceptable to apply taints AFTER registering to the cluster. Therefore, I must enrich my UserData with --register-with-taints, which makes the taints field in CreateNodeGroup completely pointless (or maybe that would cause more complications if AWS removed the taints if it wasn't supplied in the CreateNodeGroup API).