disconnected clients: Support operator manual interventions #12436
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DerekStrickland
force-pushed
the
f-manual-interventions-disconnected-clients
branch
from
e123452 to
a2e2a3c
Compare
DerekStrickland
force-pushed
the
f-manual-interventions-disconnected-clients
branch
from
a2e2a3c to
38ebd95
Compare
DerekStrickland
changed the title
tgross
reviewed
Apr 5, 2022
DerekStrickland
force-pushed
the
f-manual-interventions-disconnected-clients
branch
from
dc7576a to
b9d6e83
Compare
tgross
approved these changes
Apr 5, 2022
DerekStrickland
force-pushed
the
f-disconnected-client-allocation-handling
branch
from
ff9ed55 to
6791147
Compare
* Add merge helper for string maps * structs: add statuses, MaxClientDisconnect, and helper funcs * taintedNodes: Include disconnected nodes * upsertAllocsImpl: don't use existing ClientStatus when upserting unknown * allocSet: update filterByTainted and add delayByMaxClientDisconnect * allocReconciler: support disconnecting and reconnecting allocs * GenericScheduler: upsert unknown and queue reconnecting Co-authored-by: Tim Gross <[email protected]>
* api: Add struct, conversion function, and tests * TaskGroup: Add field, validation, and tests * diff: Add diff handler and test * docs: Update docs
* structs: Add alloc.Expired & alloc.Reconnected functions. Add Reconnect eval trigger by. * node_endpoint: Emit new eval for reconnecting unknown allocs. * filterByTainted: handle 2 phase commit filtering rules. * reconciler: Append AllocState on disconnect. Logic updates from testing and 2 phase reconnects. * allocs: Set reconnect timestamp. Destroy if not DesiredStatusRun. Watch for unknown status.
* TaskGroup: Validate that max_client_disconnect and stop_after_client_disconnect are mutually exclusive.
DerekStrickland
force-pushed
the
f-manual-interventions-disconnected-clients
branch
from
85b815f to
480b973
Compare
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR ensures that operator interventions
node drainandstop -purgeare correctly handled for disconnected clients.allocSetfilterByFailedReconnect. It filters allocations into a set that have failed on the client but do not have a terminal status at the server so that they can used by thereconciler.filterByTaintedhas been updated to considerDesiredStatusnow as well asClientStatus.DesiredStatusis run.reconnectingso that thereconcilercan handle purges and drains.reconnectedandexpiredhas been consolidated to in one place at the beginning of the filterto make expression and enforcement of the business rules more clear.
reconcilercomputeStophas been updated to always mark failed reconnects as stop even if the calculated number to remove is <= 0.computePlacementshas been updated to discount failed reconnects when calculating the existing allocs.computeStopByReconnectinghas been updated to add failed reconnects to the stop set if the number to remove calculated bycomputeStopis > 0.computeReconnectinghas been updated to not add failed reconnects to thereconcilerResult.allocWatcherShutdownwas removed from theReconnectfunction. Now that thereconcilerissetting the
DesiredStatusto stop, it is necessary for the runner to continue to run so that the existing syncingprocess can proceed as the rest of the logic expects.
Node.UpdateAllocDesiredTransition.Migrate= false in the case of drains and purges while the alloc was unknown.