-
Notifications
You must be signed in to change notification settings - Fork 880
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Add edge case handling to traffic routing #1190
fix: Add edge case handling to traffic routing #1190
Conversation
Signed-off-by: zezaeoh <[email protected]>
Signed-off-by: zezaeoh <[email protected]>
rollout/trafficrouting.go
Outdated
} else if c.pauseContext.IsAborted() { | ||
// when promote aborted. desired canary weight should be 0 | ||
} else if c.newRS.Status.AvailableReplicas == 0 { | ||
// when newRS available replicas num is 0. never weight to canary |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change is significant, and a surprising oversight if we are not handling this correctly. I will take some time to understand the implications of the change.
Looks like we have a few tests (both unit and e2e) that are asserting the opposite of this change. So at the very least the existing tests would need to change. But we would also want new tests to verify this change. This is assuming the change is correct -- I still need to play around and what the behavior should be vs. what is actually happening. |
Signed-off-by: zezaeoh <[email protected]>
Signed-off-by: zezaeoh <[email protected]>
Kudos, SonarCloud Quality Gate passed! 0 Bugs No Coverage information |
Hi, @jessesuen I would like to cover next cases with this changes
} else if c.pauseContext.IsAborted() {
// when promote aborted. desired canary weight should be 0
} ... when promotion is aborted, controller scale down canary replicas before change canary weight to 0 atDesiredReplicaCount := replicasetutil.AtDesiredReplicaCountsForCanary(c.rollout, c.newRS, c.stableRS, c.otherRSs)
if !atDesiredReplicaCount {
// Use the previous weight since the new RS is not ready for a new weight
for i := *index - 1; i >= 0; i-- {
step := c.rollout.Spec.Strategy.Canary.Steps[i]
if step.SetWeight != nil {
desiredWeight = *step.SetWeight
break
}
}
} ... this behavior leads some requests drop
} else if c.newRS == nil || c.newRS.Status.AvailableReplicas == 0 {
// when newRS available replicas num is 0. never weight to canary
} ... Suppose we have a rollout resource with a canary strategy like this: canary:
steps:
- setWeight: 20
- pause: {}
- setWeight: 40
- pause: {}
- setWeight: 60
- pause: {}
- setWeight: 100
- pause: {} when promotion is aborted, then canary service's endpoint will be empty. this behavior leads requests drop |
Codecov Report
@@ Coverage Diff @@
## master #1190 +/- ##
==========================================
+ Coverage 81.01% 81.05% +0.03%
==========================================
Files 105 105
Lines 9404 9459 +55
==========================================
+ Hits 7619 7667 +48
- Misses 1265 1270 +5
- Partials 520 522 +2
Continue to review full report at Codecov.
|
Thanks for the explanation. The change makes sense now and looks like all tests are passing. |
Checklist:
"fix(controller): Updates such and such. Fixes #1234"
.