Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: unavailable stable RS was not scaled down to make room for canary #739

Merged
merged 2 commits into from
Sep 28, 2020
Merged

fix: unavailable stable RS was not scaled down to make room for canary #739

merged 2 commits into from
Sep 28, 2020

Conversation

jessesuen
Copy link
Member

@jessesuen jessesuen commented Sep 27, 2020 •
edited
Loading

This is an alternative fix for the issue described in #735

Scenario:

  1. Apply a "bad" rollout (with or without steps) with an image which will crashloopbackoff, and the following surge/unavailable:
spec:
  replicas: 4
  strategy:
    canary:
      maxSurge: 1
      maxUnavailable: 0
  1. Update rollout with a new spec which will not crash
  2. Rollout will surge up, such that there are 5 total replicas (1 new, 4 old)
  3. Currently, rollout will now be stuck forever never being able to reach the second revision, because the controller decides it cannot scale down the stableRS (even when it should).

The correct behavior should be that the Rollout scales down one stableRS pod, in order to make room for another canary

The fix is that when calculating the replica counts when scaling down the stableRS, it should consider the Spec.Replicas when it is higher than Status.AvailableReplicas.

This also adds many e2e tests to verify scaling behavior

@codecov-commenter
Copy link

codecov-commenter commented Sep 27, 2020 •
edited
Loading

Codecov Report

Merging #739 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #739   +/-   ##
=======================================
  Coverage   82.57%   82.57%           
=======================================
  Files          95       95           
  Lines        8012     8013    +1     
=======================================
+ Hits         6616     6617    +1     
  Misses        994      994           
  Partials      402      402
Impacted Files Coverage Δ
utils/replicaset/canary.go 78.04% <100.00%> (+0.13%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b5aa6d7...7df4814. Read the comment docs.

@jessesuen jessesuen mentioned this pull request Sep 27, 2020
Closed
@jessesuen jessesuen force-pushed the unavailable-scaledown-fix branch from 85aa246 to 8142d2a Compare September 27, 2020 23:53
jessesuen
jessesuen commented Sep 28, 2020
View reviewed changes
utils/replicaset/canary.go
@@ -160,23 +160,23 @@ func CalculateReplicaCountsForCanary(rollout *v1alpha1.Rollout, newRS *appsv1.Re
}
}

minAvailableReplicaCount := rolloutSpecReplica - MaxUnavailable(rollout)
if GetReplicaCountForReplicaSets(oldRSs) > 0 {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this was changed from GetAvailableReplicaCountForReplicaSets to GetReplicaCountForReplicaSets to avoid the same problem with older RSs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also to simplify logic, i move the check to return early if there are older RSs up here, so all the logic below only has to deal with new and stable RS, and not have to factor in number of older RSs.

jessesuen
jessesuen commented Sep 28, 2020
View reviewed changes
utils/replicaset/canary.go

if newRS != nil && *newRS.Spec.Replicas > desiredNewRSReplicaCount && scaleDownCount > 0 {
if newRS != nil && *newRS.Spec.Replicas > desiredNewRSReplicaCount {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

&& scaleDownCount > 0 check is/was unnecessary because replicasToScaleDown <= minAvailableReplicaCount covers that.

danny-stytch
danny-stytch approved these changes Sep 28, 2020
View reviewed changes
Copy link

@danny-stytch danny-stytch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't really comment on the e2e tests, but the actual code change LGMT.

khhirani
khhirani approved these changes Sep 28, 2020
View reviewed changes
Copy link
Contributor

@khhirani khhirani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

dthomson25
dthomson25 approved these changes Sep 28, 2020
View reviewed changes
Copy link
Member

@dthomson25 dthomson25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What @danny-stytch said.

@jessesuen jessesuen merged commit b221be0 into argoproj:master Sep 28, 2020
jessesuen added a commit that referenced this pull request Sep 28, 2020
@jessesuen
fix: unavailable stable RS was not scaled down to make room for canary (
8239dd7 
#739)
This was referenced Oct 15, 2020
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@danny-stytch danny-stytch danny-stytch approved these changes

@khhirani khhirani khhirani approved these changes

@dthomson25 dthomson25 dthomson25 approved these changes

Assignees

@khhirani khhirani

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jessesuen @codecov-commenter @dthomson25 @khhirani @danny-stytch