Expander priority uses AWS auto-scaling groups instead of node group names

[balusarakesh]

Which component are you using?:
Expander
cluster-autoscaler

chart version: 9.4.0
image tag: v1.18.1

Component version:

kubectl version Output

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:25:00Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.12-eks-7684af", GitCommit:"7684af4ac41370dd109ac13817023cb8063e3d45", GitTreeState:"clean", BuildDate:"2020-10-20T22:57:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"

What environment is this in?:
AWS EKS v1.17 (managed node groups)

What did you expect to happen?:
CA updates the auto-scaling group according to the priorites

What happened instead?:
CA is not recognizing the node groups with their names but instead it is using AWS auto-scaling group names and falling back to picking a random node group.

How to reproduce it (as minimally and precisely as possible):
Deploy the latest CA helm chart in an AWS environment with EKS clusters and set expander to priority

we use AWS EKS node groups, CA seems to use the AWS auto-scaling group (example: eks-40bbb26b-8679-eb64-d33a-4ba475413529) instead of node group names (example: app-standard-spot-a-qa-5bkjm)
expander config:

apiVersion: v1
kind: ConfigMap
data:
  priorities: |-
    20:
      - .*spot.*
    10:
      - .*on-demand.*

logs:

Priority expander: node group eks-40bbb26b-8679-eb64-d33a-4ba475413529 not found in priority expander configuration. The group won't be used

eks-40bbb26b-8679-eb64-d33a-4ba475413529 is actually an auto-scaling group and NOT a node group. Node groups are named something like app-standard-spot-a-qa-5bkjmt

[burkel24]

I am also experiencing this issue. Happy to provide any additional diagnostic info as needed.

[cablespaghetti]

I've also got this issue and as the cluster autoscaler has no knowledge of managed node groups, and the priority expander always works on the name of the node group (in this case autoscaling group name) for all cloud providers, I don't think it's likely to be implemented in the autoscaler itself.

The conclusion I've come to is that I need to write a little application to generate the priority expander ConfigMap by finding out the ASG details from the managed node group API. When I get something working I'll post it here.

[shelby-moore]

Just ran in to this as well, I opened an issue on the AWS containers roadmap to be able to set the name or at least the prefix for the name of the underlying ASGs for Managed Node Groups.

aws/containers-roadmap#1304

[cablespaghetti]

Built a tool to work around this issue: https://github.com/cablespaghetti/priority-expander-eks-managed-nodegroup-configurer

It's very much an MVP at the moment and could use some more testing/error handling. However it works!

edit: Having an intermittent issue. See: #3956

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[runningman]

For anyone else landing here, it was fixed in eks 1.21
https://aws.amazon.com/blogs/containers/amazon-eks-1-21-released/#:~:text=Cluster%20Autoscaler%20priority%20expander%20for%20managed%20node%20groups