provide -no-shutdown-delay flag for job/alloc stop
#11596
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tgross
force-pushed
the
f-immediate-stop
branch
from
45c0964 to
a93869e
Compare
tgross
force-pushed
the
f-immediate-stop
branch
from
a93869e to
1647bde
Compare
tgross
force-pushed
the
f-immediate-stop
branch
from
1647bde to
6d23598
Compare
tgross
force-pushed
the
f-immediate-stop
branch
from
6d23598 to
fbe74fb
Compare
tgross
force-pushed
the
f-immediate-stop
branch
from
fbe74fb to
5ae23a9
Compare
tgross
force-pushed
the
f-immediate-stop
branch
from
5ae23a9 to
85b2263
Compare
|
|
tgross
force-pushed
the
f-immediate-stop
branch
from
9e7cede to
bea3bb4
Compare
tgross
force-pushed
the
f-immediate-stop
branch
from
b99b090 to
0996798
Compare
tgross
force-pushed
the
f-immediate-stop
branch
from
0996798 to
2173424
Compare
tgross
force-pushed
the
f-immediate-stop
branch
from
2173424 to
2cd4362
Compare
Some operators use very long group/task `shutdown_delay` settings to safely drain network connections to their workloads after service deregistration. But during incident response, they may want to cause that drain to be skipped so they can quickly shed load. Provide a `-no-shutdown-delay` flag on the `nomad alloc stop` and `nomad job stop` commands that bypasses the delay. This sets a new desired transition state on the affected allocations that the allocation/task runner will identify during pre-kill on the client. Note (as documented here) that using this flag will almost always result in failed inbound network connections for workloads as the tasks will exit before clients receive updated service discovery information and won't be gracefully drained.
tgross
force-pushed
the
f-immediate-stop
branch
from
2cd4362 to
179b60b
Compare
tgross
requested review from
jrasell and
DerekStrickland
and removed request for
angrycub and
jrasell
DerekStrickland
approved these changes
Dec 13, 2021
| @@ -1600,7 +1600,7 @@ func (s *StateStore) upsertJobImpl(index uint64, job *structs.Job, keepVersion b | |||
| } | |||
|
|
|||
| if err := s.updateJobCSIPlugins(index, job, existingJob, txn); err != nil { | |||
| return fmt.Errorf("unable to update job scaling policies: %v", err) | |||
| return fmt.Errorf("unable to update job csi plugins: %v", err) | |||
There was a problem hiding this comment.
This is intentional for this change set?
There was a problem hiding this comment.
Yeah, just a fixing a nearby typo while I was here.
Co-authored-by: Derek Strickland <[email protected]>
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #11448
Some operators use very long group/task
shutdown_delaysettings tosafely drain network connections to their workloads after service
deregistration. But during incident response, they may want to cause
that drain to be skipped so they can quickly shed load.
Provide a
-no-shutdown-delayflag on thenomad alloc stopandnomad job stopcommands that bypasses the delay. This sets a newdesired transition state on the affected allocations that the
allocation/task runner will identify during pre-kill on the client.
Note (as documented here) that using this flag will almost always
result in failed inbound network connections for workloads as the
tasks will exit before their clients receive updated service discovery
information and won't be gracefully drained.