constraintenforcer: Trigger task restarts when appropriate #1958
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #1958 +/- ##
==========================================
+ Coverage 54.5% 54.58% +0.08%
==========================================
Files 108 108
Lines 18577 18586 +9
==========================================
+ Hits 10125 10146 +21
+ Misses 7217 7205 -12
Partials 1235 1235Continue to review full report at Codecov.
|
dongluochen
reviewed
Feb 15, 2017
| // will bypass actions such as | ||
| // restarting the task on another node | ||
| // (if applicable). | ||
| t.Status.State = api.TaskStateRejected |
There was a problem hiding this comment.
Change function name from shutdownNoncompliantTasks to rejectNoncompliantTasks?
|
LGTM |
The constraint enforcer currently sets task desired state to "shutdown" directly, which means the orchestrator will consider these tasks already processed, and won't trigger restarts. While this is appropriate for global services, replicated services should keep the desired number of replicas running, so each task shut down by the constraint enforcer should be restarted somewhere else. This changes the constraint enforcer to trigger a task shutdown by updating the actual state rather than desired state. This will cause the orchestrator to restart the task when necessary. It's not a perfect solution, because it bends rules about field ownership, and may cause the replacement task to start before the old one stops. However, it's a good compromise solution for this problem that doesn't require absorbing the constraint enforcer into each orchestrator (which wouldn't fit the model well), or adding a third type of state to every task. Also, update the global orchestrator to only restart a task when the node still meets the constraints. Signed-off-by: Aaron Lehmann <[email protected]>
aaronlehmann
force-pushed
the
constraint-enforcer-task-replacement
branch
from
8c896af
to
69fcc19
Compare
|
LGTM |
This was referenced Feb 18, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The constraint enforcer currently sets task desired state to "shutdown"
directly, which means the orchestrator will consider these tasks already
processed, and won't trigger restarts. While this is appropriate for
global services, replicated services should keep the desired number of
replicas running, so each task shut down by the constraint enforcer
should be restarted somewhere else.
This changes the constraint enforcer to trigger a task shutdown by
updating the actual state rather than desired state. This will cause the
orchestrator to restart the task when necessary. It's not a perfect
solution, because it bends rules about field ownership, and may cause
the replacement task to start before the old one stops. However, it's a
good compromise solution for this problem that doesn't require absorbing
the constraint enforcer into each orchestrator (which wouldn't fit the
model well), or adding a third type of state to every task.
Addresses moby/moby#31014
cc @dongluochen @aluzzardi