-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ws-manager: Allow extra time for workspace to receive SIGKILL #14498
Conversation
started the job as gitpod-build-to-grace.1 because the annotations in the pull request description changed |
Volumes: volumes, | ||
RestartPolicy: corev1.RestartPolicyNever, | ||
Volumes: volumes, | ||
TerminationGracePeriodSeconds: &graceSec, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤯 this is a good idea, I think. For example, Kubernetes even recommends to do exactly this (like when saving content for a stopping container).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI @utam0k tests are currently failing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@aledbf this seems like a good idea, to add a grace period. Can you think of a reason not to add?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm also wondering if I missed something important and haven't fixed the test. I will first listen to your opinions and if it looks good, I will fix the test. That's why I chose Draft.
/werft run with-integration-tests=workspace 👍 started the job as gitpod-build-to-grace.3 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea indeed.
As a follow up PR: can we detect somehow if that grace period is enough? 🤔 Can we maybe allow to configure it via gitpod.yml?
Different workloads might require different term grace period.
We had a PR before but not merged. We could consider revisiting it. But as a user, to be honest, I don't know how much time the graceful period is needed. |
Description
By default, pod accepts SIGKILL 30 seconds after receiving SIGTERM.
This means that workspace processes must be processed for termination within 30 seconds.
This may be too short for large software such as dockerd, which will clean up the overlay directory and other files when it receives a SIGTERM. (maybe)
https://github.com/moby/moby/blob/0910306bf970603ce787466a98e4294ba81af841/cmd/dockerd/trap/trap.go#L41
If the cleanup is not done in time within 30 seconds, the container's overlay directory will remain uncluttered and causes #11183
Therefore, set this to 3 minutes, which gives you a little more time. There is currently no basis for this number, so we are looking for great ideas.
Related Issue(s)
Mitigation #11183
How to test
Please give me the idea
Release Notes
Documentation
Werft options:
If enabled this will build
install/preview
Valid options are
all
,workspace
,webapp
,ide