Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(aws-eks): kubectlProvider can run out of memory #11787

Closed
winjer opened this issue Nov 30, 2020 · 2 comments · Fixed by #11957
Closed

(aws-eks): kubectlProvider can run out of memory #11787

winjer opened this issue Nov 30, 2020 · 2 comments · Fixed by #11957
Assignees
@eladb
Labels
@aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service bug This issue is a bug. effort/small Small work item – less than a day of effort p1 pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. pr/work-in-progress This PR is a draft and needs further work.
Milestone
[GA] @aws-cdk/aws...

Comments

@winjer
Copy link

winjer commented Nov 30, 2020 •
edited
Loading

When using the HelmChart resource it is possible to deploy changes that cause the provider to run out of memory.

Reproduction Steps

create an eks.HelmChart for a large chart update. In my case it was to efs-provisioner.

What did you expect to happen?

It to work.

What actually happened?

Response object is too long.

        new CustomResource (/home/doug/projects/trustsrv.io/node_modules/@aws-cdk/core/lib/custom-resource.ts:115:21)
        \_ new HelmChart (/home/doug/projects/trustsrv.io/node_modules/@aws-cdk/aws-eks/lib/helm-chart.ts:104:5)
        \_ Object.exports.deployRelease (/home/doug/projects/trustsrv.io/src/core/cluster/charts.ts:68:5)
        \_ Object.exports.systemCharts (/home/doug/projects/trustsrv.io/src/core/cluster/charts.ts:104:9)
        \_ processTicksAndRejections (internal/process/task_queues.js:97:5)

In the logs for the kubectl provider lambda function i got:

fatal error: runtime: out of memory

and a whole lot of go traceback.

Environment

  • CDK CLI Version : 1.59.0
  • Framework Version: 1.59.0
  • Node.js Version: 14.2
  • OS : Debian
  • Language (Version): Typescript 3.7.3

Other

I resolved this by manually changing the memory on the kubectl provider python lambda function and giving it 1024MB, using the console.

This is 🐛 Bug Report
@winjer winjer added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Nov 30, 2020
@SomayaB SomayaB changed the title eks: kubectlProvider can run out of memory (eks): kubectlProvider can run out of memory Nov 30, 2020
@github-actions github-actions bot added the @aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service label Nov 30, 2020
@iliapolo iliapolo added effort/small Small work item – less than a day of effort p1 and removed needs-triage This issue or PR still needs to be triaged. labels Dec 1, 2020
@iliapolo iliapolo changed the title (eks): kubectlProvider can run out of memory (aws-eks): kubectlProvider can run out of memory Dec 1, 2020
This was referenced Dec 1, 2020
Closed
Closed
@iliapolo iliapolo added this to the [GA] @aws-cdk/aws-eks milestone Dec 1, 2020
@jumi-dev
Copy link
Contributor

jumi-dev commented Dec 2, 2020

I run into the same error message today ("Response object is too long") when I deployed a larger Kubernetes Manifests using eks.addManifest.

It should be the same issue. In both cases, the deployment is done using the kubectl provider lambda.

Maybe this information is relevant if you do some tests etc.

@eladb eladb assigned eladb and unassigned iliapolo Dec 9, 2020
eladb pushed a commit that referenced this issue Dec 9, 2020
fix(eks): kubectl provider out-of-memory for large manifests/charts
d36297c 
Increase the default memory size of the kubectl provider's lambda function to 1GiB and introduce a `kubectlMemory` option that can be used to control memory allocation if needed.

Fixes #11787
@eladb eladb mentioned this issue Dec 9, 2020
Merged
@eladb eladb added pr/work-in-progress This PR is a draft and needs further work. pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. labels Dec 9, 2020
@mergify mergify bot closed this as completed in #11957 Dec 9, 2020
mergify bot pushed a commit that referenced this issue Dec 9, 2020
fix(eks): kubectl provider out-of-memory for large manifests/charts (…
2ec2948 
…now 1GiB) (#11957)

Increase the default memory size of the kubectl provider's lambda function to 1GiB and introduce a `kubectlMemory` option that can be used to control memory allocation if needed.

Fixes #11787


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

github-actions bot commented Dec 9, 2020

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

flochaz pushed a commit to flochaz/aws-cdk that referenced this issue Jan 5, 2021
@flochaz
fix(eks): kubectl provider out-of-memory for large manifests/charts (…
b296181 
…now 1GiB) (aws#11957)

Increase the default memory size of the kubectl provider's lambda function to 1GiB and introduce a `kubectlMemory` option that can be used to control memory allocation if needed.

Fixes aws#11787


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@eladb eladb

Labels
@aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service bug This issue is a bug. effort/small Small work item – less than a day of effort p1 pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. pr/work-in-progress This PR is a draft and needs further work.
Projects
None yet
Milestone
[GA] @aws-cdk/aws-eks
Development

Successfully merging a pull request may close this issue.

fix(eks): kubectl provider out-of-memory for large manifests/charts (now 1GiB) aws/aws-cdk
4 participants
@winjer @eladb @iliapolo @jumi-dev