Cluster Autoscaler does not have permissions to list replicasets.apps

[ofaz]

When deploying the cluster autoscaler as per: https://eksworkshop.com/scaling/deploy_ca/, errors are generated suggesting that the cluster autoscaler cannot list replicasets.apps. I believe in some cases this also prevents any scale down or scale up activity from successfully occurring.

The error logs take the form of:
Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kube-system:cluster-autoscaler"

A similar issue was found with other platforms deployments of the cluster autoscaler (e.g. Azure/aks-engine#261)

Is this expected / correct behaviour? I was able to address this by modifying the cluster-autoscaler.yaml at line 44 to include replicasets in its ClusterRole permissions. (https://github.com/aws-samples/eks-workshop/blob/master/content/scaling/deploy_ca.files/cluster_autoscaler.yml#L44)

Happy to submit a PR if this is a confirmed issue!

[malz]

eks-workshop/content/scaling/deploy_ca.files/cluster_autoscaler.yml

Lines 37 to 39 in 3ec6e6a

	- apiGroups: ["extensions"]
	resources: ["replicasets","daemonsets"]
	verbs: ["watch","list","get"]

It would appear that replicasets has been moved in apigroups from "extensions" (I believe when it was still in v1beta?) to "apps" permanently as of Kubernetes 1.14

See https://kubernetes.io/docs/setup/release/notes/#deprecations under API

DaemonSet, Deployment, and ReplicaSet resources will no longer be served from extensions/v1beta1, apps/v1beta1, or apps/v1beta2 in v1.16. Migrate to the apps/v1 API, available since v1.9. Existing persisted data can be retrieved via the apps/v1 API.

As EKS only supports from 1.10 - 1.12 I think for future compatibility (and correctness sake) that replicasets is moved from extensions to apps. This will be more or less an invisible change to the user.

[Jeffwan]

Permissions has been fixed in

kubernetes/autoscaler#2229
kubernetes/autoscaler#2230
kubernetes/autoscaler#2231

We can close this issue.

[malz]

Will the upstream fixes work if the user is following this tutorial? It would require an update to cluster autoscaler correct? Otherwise the solution posted above would fix it for this tutorial.

[Jeffwan]

Ah. I didn't notice that. Yeah. We need to submit a PR to this repo to use examples from CA aws examples.

[toricls]

I believe this should be fixed with https://github.com/aws-samples/eks-workshop/blob/master/content/beginner/080_scaling/deploy_ca.files/cluster_autoscaler.yml#L44