feat: Wait for canary RS to have ready replicas before shifting labels

[aweis89]

feat: Wait for first batch of new RS pods to be ready before adding canary selector label to an SVC. Fixes #1026

• Last commit does a bit of extra refactoring for the reconcileStableAndCanaryService function (can easily be reversed if not desirable).
  • Moves a bit of duplicated logic, previously executed for both canary and stable SVCs and RSs, into it's own function ensureSVCTargets.
• Adds a new function newRSReady which needs to return true before the canary SVC gets the newRS selector label.
  • The newRS needs to exist, not have ReadyReplicas < Replicas and not have 0 ReadyReplicas, thereby ensuring the first batch of Pods are ready before targeting the newRS.

(Was also considering perhaps making this label addition an optional step so when specified, the user can control precisely when the label swap happens.)

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
• The title of the PR is (a) conventional, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
• I've signed my commits with DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My builds are green. Try syncing with master if they are not.
• My organization is added to USERS.md.

[codecov-io]

Codecov Report

Merging #1022 (76d65cb) into master (6f501f7) will increase coverage by 0.03%.
The diff coverage is 77.77%.

@@            Coverage Diff             @@
##           master    #1022      +/-   ##
==========================================
+ Coverage   80.90%   80.93%   +0.03%     
==========================================
  Files         102      102              
  Lines        9080     9086       +6     
==========================================
+ Hits         7346     7354       +8     
+ Misses       1240     1239       -1     
+ Partials      494      493       -1     

Impacted Files	Coverage Δ
rollout/service.go	78.94% <77.77%> (+4.66%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6f501f7...76d65cb. Read the comment docs.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[jessesuen]

Great improvement!