Not discovering ASGs on AWS

[MartinEmrich]

Hello!

I am trying to get autoscaler to work, I use v 1.3.7 (which should match with AWS's K8s 1.11).

I have 3 ASGs (one for each AZ), all tagged:

k8s.io/cluster-autoscaler/awsref | owned  
k8s.io/cluster-autoscaler/enabled | true  

Here's my startup parameters:

          command:
            - ./cluster-autoscaler
            - --v=99
            - --stderrthreshold=info
            - --cloud-provider=aws
            - --skip-nodes-with-local-storage=false
            - --expander=least-waste
            - >-
              --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,
              k8s.io/cluster-autoscaler/awsref

But I get this message if I try to create more pods to trigger a scale-up:

  Normal   NotTriggerScaleUp  2m19s (x10 over 3m57s)  cluster-autoscaler  pod didn't trigger scale-up (it wouldn't fit if a new node is added)

In the autoscaler logs, I see just this:

I0403 09:12:58.999402       1 static_autoscaler.go:135] Starting main loop
I0403 09:12:59.052834       1 auto_scaling_groups.go:316] Regenerating instance to ASG map for ASGs: []
I0403 09:12:59.052851       1 aws_manager.go:148] Refreshed ASG list, next refresh after 2019-04-03 09:13:09.052847656 +0000 UTC m=+1980.976476835

I would assume that if autoscaler would detect my ASGs, they would appear between the square braces?

Both the EKS cluster and the workers have the prescribed IAM permissions.

[MartinEmrich]

Additional info: I see no API calls for Autoscaling Groups in AWS CloudTrail, so either they fail or are not made at all.

[Jeffwan]

@MartinEmrich Change to this and have a another try,

- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/awsref

BTW, I would suggest you to also include cluster tag. Otherwise, your ASG maybe detected by different clusters if you create multiple cluster in your aws account.

- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/benchmark-0403

I do think samples has been edited and cause this issue. I will submit a PR to fix this.

[Jeffwan]

@audrey2424 ???

[MartinEmrich]

@Jeffwan the first change did work:

I0404 09:13:08.682572       1 auto_scaling_groups.go:316] Regenerating instance to ASG map for ASGs: [eksctl-awsref-nodegroup-awsref-a-1-NodeGroup-AQ42UG0E1F09 eksctl-awsref-nodegroup-awsref-b-1-NodeGroup-PAHLMNQBO5ID eksctl-awsref-nodegroup-awsref-c-1-NodeGroup-1N8YHNIEZEPDQ]

All it changed was really the YAML formatting (with the >- thingy), which apparently was not YAML, but just a typo in the example?... Diff:

             - --expander=least-waste
-            - >-
-              --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,
-              k8s.io/cluster-autoscaler/awsref
+            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/awsref
           volumeMounts:

I'm unsure about "I would suggest you to also include cluster tag.". Isn't k8s.io/cluster-autoscaler/awsref my cluster tag already? What is "benchmark-0403"?

[MartinEmrich]

FYI: I just did some tests (adding a ***load of pods, and later removing them), cluster-autoscaler works great! 👍

[michaa76]

@MartinEmrich would you please close the issue, if it works for you?!

[Jeffwan]

I'm unsure about "I would suggest you to also include cluster tag.". Isn't k8s.io/cluster-autoscaler/awsref my cluster tag already? What is "benchmark-0403"?

No, that's not. If you look at the documentation here.
https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/aws#auto-discovery-setup

You know what I mean. Sorry benchmark-0403 is my cluster name, you need to replace with your cluster name to limit CA scope. You don't like CA to fetch other cluster's ASGs.

[MartinEmrich]

@Jeffwan

The docs say:

It is recommended to use a second tag like k8s.io/cluster-autoscaler/YOUR CLUSTER NAME when k8s.io/cluster-autoscaler/enabled is used across many clusters to prevent ASGs from different clusters recognized as the node groups

as "awsref" is my cluster name, that should be the cluster tag? I am confused.

[Jeffwan]

@MartinEmrich I see. My careless. Did not notice that's a cluster name. It should be fine. If you solve the problem, we can close issue. :D