Automatic deployment of Cluster Autoscaler

[filiphaftek]

I have issues

Although worker nodes are deployed as autoscaling group, when EKS cannot schedule more pods because of missing resources i.e. CPU more nodes are not started by ASG. Would be nice to deploy Cluster Autoscaler automatically (or at least put some information in README how to do this) so we can benefit from ASG.

I'm submitting a...

• feature request

What is the current behavior?

ASG workers are not started even if EKS cannot schedule more pods because of missing CPU and we are still below the maximum size of workers ASG.

What's the expected behaviour?

The expected behaviour would be:

1. deploy Cluster Autoscaler: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md
   (
   this permissions needs to be added for EC2 EKS IAM role:

"autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"

)
2. Scale sample application so it needs more than whole CPU from single VM.
3. See that Autoscaler is adding more nodes.

Environment details

ESK in us-east-1

• Terraform version:
  v0.11.7

[brandonjbjelland]

Hey @filiphaftek -

Thanks for submitting this. I'm still trying to find the right balance of which features of k8s are worth choosing as winners and losers within this module... at the moment I leave it up to the operator but it seems like enough of this sort of request have rolled in to warrant a solution.

Assumption on the implementation: it sounds like this is something we would be able to create after the EKS control plane is up through a null_resource local-exec of some kubectl. I'd much rather use the kubernetes provider in terraform but that still seems incompatible (no rolebinding). Does that seem like the right way to you? If we can hide options like this behind simple flags within the variables, I can see solid patterns emerging of conditionally adding functionality.

Thoughts on my proposed approach? Let me know if you're able to take on a PR here and I'll be happy to try and push that to the finish line.

[max-rocket-internet]

I would vote to not to include the cluster-autoscaler installation in this module. I think installing stuff into the cluster is beyond the scope of this module.

But I think we should optionally add the IAM policies to allow the cluster-autoscaler to run. I will add a PR for this soon. #78

Also, @filiphaftek I recommend you use helm to install the cluster-autoscaler 😉

[filiphaftek]

hi @max-rocket-internet
I agree that just adding the proper IAM policies and some information in README how to install Cluster Autoscaler (i.e. with helm :) might be enough.

hello @brandoconnor
would be nice to use terraform kubernetes provider when it's ready, as local-exec looks more like a workarounds than a final implementation. But as for now, proper IAM policies automatically added and even a simple README line how to deploy it should be enough.

[max-rocket-internet]

Initial PR: #93

[filiphaftek]

@max-rocket-internet thank you very much for the PR! Since version v1.5.0 autoscaling policies are in place and after deployment of AS all is working.
@brandoconnor could you close this issue as it's revolved?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.