Cluster Autoscaler on AWS is OOM killed on startup in GenerateEC2InstanceTypes

[timothyb89]

We noticed our cluster autoscaler occasionally getting OOM killed on startup or when elected as leader. The memory usage spike on startup is fairly consistent even when not OOM killed, sitting just below the default limits at 250Mi or so. When it doesn't OOM, this memory is eventually garbage collected and the autoscaler stabilizes at well under 100Mi used:

After a pprof trace (requiring an ad-hoc upgrade to cluster-autoscaler v1.18.2 to get the --profiling flag) we noticed a large chunk of memory allocated in the GenerateEC2InstanceTypes function. We were able to trace this back to PR #2249 which fetches an updated list of EC2 instance types from an AWS-hosted JSON file. Surprisingly, this file is 94 MiB, the entirety of which is fetched onto the heap before parsing. The data extracted is fairly small (under 43KiB per ec2_instance_types.go) but unfortunately the allocations sometimes live long enough to push the autoscaler over the (default) memory limit.

Additionally, with the --aws-use-static-instance-list=true flag set, the memory spike disappears:

Is there some solution that could fetch the updated list without requiring an otherwise unnecessary memory limit increase? Given the autoscaler's special priority class, raising the limit well beyond what it actually needs at runtime feels a bit wrong.

Additional information:

• autoscaler image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.16.6

• Kubernetes version:

  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-eks-2ba888", GitCommit:"2ba888155c7f8093a1bc06e3336333fbdb27b3da", GitTreeState:"clean", BuildDate:"2020-07-17T18:48:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  

• pprof svg: cluster-autoscaler-pprof.tar.gz (.svg in a tarball to satisfy GitHub)

• kubectl describe pod output:

  Name:                 cluster-autoscaler-7b9c56647d-9v8pr
  Namespace:            kube-system
  Priority:             2000000000
  Priority Class Name:  system-cluster-critical
  Node:                 ip-192-168-125-18.us-west-2.compute.internal/192.168.125.18
  Start Time:           Tue, 08 Sep 2020 05:18:08 -0600
  Labels:               app=cluster-autoscaler
                        app.kubernetes.io/instance=cluster-autoscaler
                        app.kubernetes.io/name=cluster-autoscaler
                        pod-template-hash=7b9c56647d
  Annotations:          cluster-autoscaler.kubernetes.io/safe-to-evict: false
                        kubernetes.io/psp: psp.privileged
                        prometheus.io/path: /metrics
                        prometheus.io/port: 8085
                        prometheus.io/scrape: true
  Status:               Running
  IP:                   192.168.121.122
  IPs:
    IP:           192.168.121.122
  Controlled By:  ReplicaSet/cluster-autoscaler-7b9c56647d
  Containers:
    cluster-autoscaler:
      Container ID:  docker://cbdbb11a7c20b042d79744edbb5dd0c6fde71303be697a1a773307c9d5ac442c
      Image:         k8s.gcr.io/autoscaling/cluster-autoscaler:v1.16.6
      Image ID:      docker-pullable://k8s.gcr.io/autoscaling/cluster-autoscaler@sha256:cbbe98dd8f325bef54557bc2854e48983cfc706aba126bedb0c52d593e869072
      Port:          8085/TCP
      Host Port:     0/TCP
      Command:
        ./cluster-autoscaler
        --v=4
        --stderrthreshold=info
        --cloud-provider=aws
        --skip-nodes-with-local-storage=false
        --expander=least-waste
        --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<snip>1
        --balance-similar-node-groups
        --skip-nodes-with-system-pods=false
      State:          Running
        Started:      Wed, 09 Sep 2020 16:56:37 -0600
      Last State:     Terminated
        Reason:       OOMKilled
        Exit Code:    137
        Started:      Wed, 09 Sep 2020 16:52:52 -0600
        Finished:     Wed, 09 Sep 2020 16:56:21 -0600
      Ready:          True
      Restart Count:  2
      Limits:
        cpu:     100m
        memory:  300Mi
      Requests:
        cpu:        100m
        memory:     300Mi
      Environment:  <none>
      Mounts:
        /etc/ssl/certs/ca-certificates.crt from ssl-certs (ro)
        /var/run/secrets/kubernetes.io/serviceaccount from cluster-autoscaler-token-bwpc6 (ro)
  Conditions:
    Type              Status
    Initialized       True 
    Ready             True 
    ContainersReady   True 
    PodScheduled      True 
  Volumes:
    ssl-certs:
      Type:          HostPath (bare host directory volume)
      Path:          /etc/ssl/certs/ca-bundle.crt
      HostPathType:  
    cluster-autoscaler-token-bwpc6:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  cluster-autoscaler-token-bwpc6
      Optional:    false
  QoS Class:       Guaranteed
  Node-Selectors:  <none>
  Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                   node.kubernetes.io/unreachable:NoExecute for 300s
  Events:          <none>
  

[hhamalai]

Experienced the same OOM issue when disabling IMDSv1 and switching purely to use IRSA, but the deployment was missing AWS_REGION environment variable, which leads Cluster Autoscaler to query the pricing information for all available regions. With these JSON document sizes, OOMKills are likely to happen. With AWS_REGION specified, only matching pricing data will be retrieved

autoscaler/cluster-autoscaler/cloudprovider/aws/aws_util.go

Line 71 in d054bb2

if region != "" && region != r.ID() {

[seamusabshere]

I am seeing this error even with 6Gi of memory limits... something is wrong.

[mcristina422]

We've seen similar issues with our AWS autoscaler. We didn't have 1.18 to take a pprof but it was taking > 5GB of ram. Maybe we should default to the static list? I don't think it's been updated recently

[seamusabshere]

I cross-commented on this related issue: #3044 (comment)

[jaypipes]

I cross-commented on this related issue: #3044 (comment)

I actually don't believe the issues #3044 and this one are due to the same problem. This one is pretty clearly the result of the instance type dynamic generation pulling down 100+MB JSON files on startup. In #3044, however, yourself and another poster point out that using the static instance type list does not solve memory leak issues. I believe the root cause of these issues is different.

[jqmichael]

If adding AWS_REGION to container could fix pulling down huge instance type file, it was added to the latest https://github.com/aws/amazon-eks-pod-identity-webhook and also enabled in 1.18 EKS cluster.

[seamusabshere]

[ @jaypipes means #3044 not #3004 above ]

[jaypipes]

[ @jaypipes means #3044 not #3004 above ]

doh, yep, sorry about that! fixed :)

[ellistarn]

I'm trying to reproduce. How big of a cluster are you trying out? I'm running 1.18.2 on EKS 1.18. 100 node cluster, 400 pods and it's sitting stable at 300mb of memory.

[ellistarn]

Update after deep diving this. @seamusabshere's OOM was due to listwatch caches filling up on startup due to a large number of Job objects in the API Server.

@timothyb89, is there any chance your cluster is suffering similar fate?

[seamusabshere]

# courtesy https://stackoverflow.com/a/61231027/310192
kubectl delete jobs --field-selector status.successful=1 

😆

i thought i was safe because we were using ttlSecondsAfterFinished... but that's an alpha feature and per @ellistarn "[EKS runs] feature gates that are in Beta."

So, I had thousands of months-old jobs.

[timothyb89]

@timothyb89, is there any chance your cluster is suffering similar fate?

Our largest cluster has 250 job objects at the moment, which I'd hope isn't nearly large enough to cause any trouble.

For what it's worth we been using --aws-use-static-instance-list=true since September and have not seen any unexpected restarts.

[e-nalepa]

I reproduce the issue, even if contrary to the initial ticket, the default limit it now not set at 250 Mi but at 300 Mi.

Sometimes, the cluster-autoscaler pod needs a lot more, like bellow (572 Mi):

NAME                                        CPU(cores)   MEMORY(bytes)
...
aws-node-m548b                              4m           40Mi
cluster-autoscaler-6478668dc5-j6gql         93m          572Mi
coredns-6d97dc4b59-dc72r                    3m           7Mi
...

Increasing this limit accordingly solves the issue on my side.

[carlosjgp]

Is there any chance we could grab this list from the local filesystem and in combination with using an initContainer or a static configmap we could contain the "controller" memory limits closer to the requests?

We have to configure requests: 96Mi and limits: 512Mi... I bet that list is going to keep growing and eventually crash the PODs 😓

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[carlosjgp]

/remove-lifecycle stale

[jolestar]

Fixed by increment the memory limit to 800Mi

[mindw]

Fixed by increment the memory limit to 800Mi

That option was mentioned and referenced above several times. #3506 (comment) Idea of having the list provided as input is IMHO one of the better options.

[zerthimon]

Having the same issue with 1.21.0, downgrading to 1.20.0 fixes the problem.
Edit: increasing memory limit of the pod also fixes the issue.
The process takes ~500MB resident memory. Does it really need so much memory to work ?

[Zauxst]

I am suffering from this problem as well on AWS EKS... It started with 1.19.1; I've upgraded to 1.20.0 and now I'm looking at upgrading the memory.
We have default configurations.

[watkinsmike]

This is also affecting me, although for some reason only on a single cluster, even using the exact same configuration and limits. Upping the mem limits to 500 has been the workaround I used successully

[nakamasato]

This issue seems to be mitigated by #4199, but for now, it seems the only way is to increase the memory requests and limits like #4207

[fblgit]

we also have this issue, even up to 1G.. stills dying... somewhere there is a leak and is related with CA & AWS.

@gjtempleton do u guys have a date for the #4199 milestone ? in which release we can expect this?

[gjtempleton]

Currently #4199 has made it into the default branch and been cherry-picked back to the 1.19, 1.20, 1.21 and 1.22 release branches, so will make it into the next patch releases of all those versions. #4251 is the issue to watch for the cutting of those releases.

[kragniz]

I've deployed v1.22.1 into a cluster which was previously seeing an oomkill with a memory limit of 300Mi. It's fixed the problem for us.

[gjtempleton]

Awesome, thanks for letting us know, all credit to @aidy for doing the hard work.

I'll leave this issue open for a bit longer to see if anyone's still seeing these issues with the new patch releases that include the streaming change, but if not will close it off in a week.

[gjtempleton]

Given the lack of any new reports of this issue, I'm going to close this as resolved for now, please let us know if you see any recurrence of this behaviour though.

/close

[k8s-ci-robot]

@gjtempleton: Closing this issue.

In response to this:

Given the lack of any new reports of this issue, I'm going to close this as resolved for now, please let us know if you see any recurrence of this behaviour though.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[zerthimon]

v1.22.1 fixed the issue for me.

[tarun4279]

Cluster Autoscaler internally downloads a JSON file as @hhamalai mentioned. In my case it was 114MB. By giving the cluster autoscaler pod some additional memory, the issue was fixed.

Checkout for the log fetching https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/ap-south-1/index.json

[bytewhisperer11]

I've deployed v1.22.1 into a cluster which was previously seeing an oomkill with a memory limit of 300Mi. It's fixed the problem for us.

This worked for me thanks.