[gh-6980] Client: clean up old allocs before running new ones using the exec task driver.
#20500
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Juanadelacuesta
force-pushed
the
gh-b-6980
branch
3 times, most recently
from
8a6f933
to
aaea742
Compare
Juanadelacuesta
force-pushed
the
gh-b-6980
branch
2 times, most recently
from
0ebb10b
to
603c40a
Compare
Juanadelacuesta
changed the title
exec task driver.
Juanadelacuesta
requested review from
shoenig,
tgross,
gulducat,
angrycub and
jrasell
tgross
requested changes
May 8, 2024
go.mod
Outdated
| @@ -172,7 +172,7 @@ require ( | |||
| github.com/cespare/xxhash/v2 v2.2.0 // indirect | |||
| github.com/checkpoint-restore/go-criu/v5 v5.3.0 // indirect | |||
| github.com/cheggaaa/pb/v3 v3.0.5 // indirect | |||
| github.com/cilium/ebpf v0.9.1 // indirect | |||
| github.com/cilium/ebpf v0.11.0 // indirect | |||
There was a problem hiding this comment.
I tested a few things before, this is probably a left over of one of those attempts.
There was a problem hiding this comment.
Ok, we'll want to back this change out before merging.
Co-authored-by: Tim Gross <[email protected]>
Juanadelacuesta
added
backport/1.6.x
tgross
reviewed
May 10, 2024
tgross
reviewed
May 10, 2024
There was a problem hiding this comment.
Overall this is looking great @Juanadelacuesta. I've left one question about the signals we're masking, and then there's a couple remaining bits to mop up like the go.mod and then I think this will be ready to ship.
| signal.Notify(l.sigChan, syscall.SIGHUP, syscall.SIGQUIT, syscall.SIGTERM, syscall.SIGINT, syscall.SIGSEGV) | ||
| for { | ||
| signal := <-l.sigChan | ||
| if signal == syscall.SIGTERM || signal == syscall.SIGKILL || signal == syscall.SIGINT { |
There was a problem hiding this comment.
SIGKILL isn't one of the signals we've masked above, and we can't catch it anyways. So this block really is only translating SIGTERM to SIGINT, right?
Juanadelacuesta
force-pushed
the
gh-b-6980
branch
from
e43528e
to
9d6b62e
Compare
Co-authored-by: Tim Gross <[email protected]>
This was referenced May 14, 2024
Juanadelacuesta
added a commit
that referenced
this pull request
May 14, 2024
…he `exec` task driver. (#20500) Whenever the "exec" task driver is being used, nomad runs a plug in that in time runs the task on a container under the hood. If by any circumstance the executor is killed, the task is reparented to the init service and wont be stopped by Nomad in case of a job updated or stop. This commit introduces two mechanisms to avoid this behaviour: * Adds signal catching and handling to the executor, so in case of a SIGTERM, the signal will also be passed on to the task. * Adds a pre start clean up of the processes in the container, ensuring only the ones the executor runs are present at any given time.
Juanadelacuesta
added a commit
that referenced
this pull request
May 14, 2024
…he `exec` task driver. (#20500) Whenever the "exec" task driver is being used, nomad runs a plug in that in time runs the task on a container under the hood. If by any circumstance the executor is killed, the task is reparented to the init service and wont be stopped by Nomad in case of a job updated or stop. This commit introduces two mechanisms to avoid this behaviour: * Adds signal catching and handling to the executor, so in case of a SIGTERM, the signal will also be passed on to the task. * Adds a pre start clean up of the processes in the container, ensuring only the ones the executor runs are present at any given time.
Juanadelacuesta
added a commit
that referenced
this pull request
May 15, 2024
…he `exec` task driver. (#20500) (#20584) Whenever the "exec" task driver is being used, nomad runs a plug in that in time runs the task on a container under the hood. If by any circumstance the executor is killed, the task is reparented to the init service and wont be stopped by Nomad in case of a job updated or stop. This commit introduces two mechanisms to avoid this behaviour: * Adds signal catching and handling to the executor, so in case of a SIGTERM, the signal will also be passed on to the task. * Adds a pre start clean up of the processes in the container, ensuring only the ones the executor runs are present at any given time. Co-authored-by: Juana De La Cuesta <[email protected]>
Juanadelacuesta
added a commit
that referenced
this pull request
May 15, 2024
…ones using the `exec` task driver. into release/1.6.x (#20583) * [gh-6980] Client: clean up old allocs before running new ones using the `exec` task driver. (#20500) Whenever the "exec" task driver is being used, nomad runs a plug in that in time runs the task on a container under the hood. If by any circumstance the executor is killed, the task is reparented to the init service and wont be stopped by Nomad in case of a job updated or stop. This commit introduces two mechanisms to avoid this behaviour: * Adds signal catching and handling to the executor, so in case of a SIGTERM, the signal will also be passed on to the task. * Adds a pre start clean up of the processes in the container, ensuring only the ones the executor runs are present at any given time. * fix: add the missing cgroups functions --------- Co-authored-by: Juana De La Cuesta <[email protected]> Co-authored-by: Juanadelacuesta <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Whenever the "exec" task driver is being used, nomad runs a plug in that in time runs the task on a container under the hood. If by any circumstance the executor is killed, the task is reparented to the init service and wont be stopped by Nomad in case of a job updated or stop.
This PR introduces two mechanisms to avoid this behaviour:
It fixes #6980