fix: fallback to patch on scale conflict

Signed-off-by: Zach Aller <[email protected]> fix: switch to retry logic Signed-off-by: Zach Aller <[email protected]> lint Signed-off-by: Zach Aller <[email protected]> retry experiments Signed-off-by: Zach Aller <[email protected]> remove TODO Signed-off-by: Zach Aller <[email protected]> remove accidental add Signed-off-by: Zach Aller <[email protected]> remove accidental add Signed-off-by: Zach Aller <[email protected]> add retry to setting revision Signed-off-by: Zach Aller <[email protected]> chore(deps): bump slsa-framework/slsa-github-generator from 1.10.0 to 2.0.0 (#3537) chore(deps): bump slsa-framework/slsa-github-generator Bumps [slsa-framework/slsa-github-generator](https://github.com/slsa-framework/slsa-github-generator) from 1.10.0 to 2.0.0. - [Release notes](https://github.com/slsa-framework/slsa-github-generator/releases) - [Changelog](https://github.com/slsa-framework/slsa-github-generator/blob/main/CHANGELOG.md) - [Commits](slsa-framework/slsa-github-generator@v1.10.0...v2.0.0) --- updated-dependencies: - dependency-name: slsa-framework/slsa-github-generator dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> chore(deps): bump sigstore/cosign-installer from 3.4.0 to 3.5.0 (#3522) Bumps [sigstore/cosign-installer](https://github.com/sigstore/cosign-installer) from 3.4.0 to 3.5.0. - [Release notes](https://github.com/sigstore/cosign-installer/releases) - [Commits](sigstore/cosign-installer@e1523de...59acb62) --- updated-dependencies: - dependency-name: sigstore/cosign-installer dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> chore(deps): bump golangci/golangci-lint-action from 4 to 5 (#3540) Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 4 to 5. - [Release notes](https://github.com/golangci/golangci-lint-action/releases) - [Commits](golangci/golangci-lint-action@v4...v5) --- updated-dependencies: - dependency-name: golangci/golangci-lint-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> docs: provide recommendation for strategies (#3531) * docs: provide recommendation for strategies Signed-off-by: Kostis (Codefresh) <[email protected]> * docs: traffic manager clarifications Signed-off-by: Kostis (Codefresh) <[email protected]> * docs: explain canary with/out traffic manager Signed-off-by: Kostis (Codefresh) <[email protected]> * docs: add 3 columns on the comparison table Signed-off-by: Kostis (Codefresh) <[email protected]> --------- Signed-off-by: Kostis (Codefresh) <[email protected]> feat(dashboard): change the color of the current rollout step (#3526) I feel that having the current (running) step in a orange color is misleading, as orange usually means warning. This commit changes the color to the `$argo-running-color`. Signed-off-by: Alejandro López Sánchez <[email protected]> chore(deps): bump github.com/aws/aws-sdk-go-v2/service/cloudwatch from 1.37.0 to 1.38.0 (#3525) chore(deps): bump github.com/aws/aws-sdk-go-v2/service/cloudwatch Bumps [github.com/aws/aws-sdk-go-v2/service/cloudwatch](https://github.com/aws/aws-sdk-go-v2) from 1.37.0 to 1.38.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/service/s3/v1.38.0/CHANGELOG.md) - [Commits](aws/aws-sdk-go-v2@service/s3/v1.37.0...service/s3/v1.38.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/cloudwatch dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> perform all of set revision actions on retry Signed-off-by: Zach Aller <[email protected]> fix variable Signed-off-by: Zach Aller <[email protected]> add retry counts to log Signed-off-by: Zach Aller <[email protected]> add retry counts to logs Signed-off-by: Zach Aller <[email protected]> clean logs, always dump controller e2e logs Signed-off-by: Zach Aller <[email protected]> lower timeout Signed-off-by: Zach Aller <[email protected]> bump timeout on e2e Signed-off-by: Zach Aller <[email protected]> retry on rollout conflict Signed-off-by: Zach Aller <[email protected]> don't reque on rs changes Signed-off-by: Zach Aller <[email protected]> reque rs Signed-off-by: Zach Aller <[email protected]> bump qps for e2e Signed-off-by: Zach Aller <[email protected]> fix gen-crd Signed-off-by: Zach Aller <[email protected]> switch to patch Signed-off-by: Zach Aller <[email protected]> switch to patch Signed-off-by: Zach Aller <[email protected]> add log Signed-off-by: Zach Aller <[email protected]> move log lines Signed-off-by: Zach Aller <[email protected]> Trigger Build Signed-off-by: Zach Aller <[email protected]> fix one e2e test Signed-off-by: Zach Aller <[email protected]> lint Signed-off-by: Zach Aller <[email protected]> add test Signed-off-by: Zach Aller <[email protected]> chore(deps): bump actions/setup-go from 5.0.0 to 5.0.1 (#3552) Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5.0.0 to 5.0.1. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@v5.0.0...v5.0.1) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> chore(deps): bump codecov/codecov-action from 4.3.0 to 4.3.1 (#3550) Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 4.3.0 to 4.3.1. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v4.3.0...v4.3.1) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> chore(deps): bump google.golang.org/protobuf from 1.33.0 to 1.34.0 (#3548) Bumps google.golang.org/protobuf from 1.33.0 to 1.34.0. --- updated-dependencies: - dependency-name: google.golang.org/protobuf dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> refactor Signed-off-by: Zach Aller <[email protected]> add test for updating rs revision Signed-off-by: Zach Aller <[email protected]> add retry for ephemeral metadata Signed-off-by: Zach Aller <[email protected]> clear some fields Signed-off-by: Zach Aller <[email protected]> add logs Signed-off-by: Zach Aller <[email protected]> refactor into function Signed-off-by: Zach Aller <[email protected]> change log Signed-off-by: Zach Aller <[email protected]> switch rollout update to patch fallback Signed-off-by: Zach Aller <[email protected]> siwtch ephemeral metadata sync to shared function Signed-off-by: Zach Aller <[email protected]> siwtch merge type Signed-off-by: Zach Aller <[email protected]> lint Signed-off-by: Zach Aller <[email protected]> don't update status Signed-off-by: Zach Aller <[email protected]> switch rollout update to not use patch Signed-off-by: Zach Aller <[email protected]> change log Signed-off-by: Zach Aller <[email protected]> switch to small patch Signed-off-by: Zach Aller <[email protected]> some cleanup Signed-off-by: Zach Aller <[email protected]> remove not found rollout removal Signed-off-by: Zach Aller <[email protected]> working setup Signed-off-by: Zach Aller <[email protected]> lint Signed-off-by: Zach Aller <[email protected]> fix test Signed-off-by: Zach Aller <[email protected]> small cleanup Signed-off-by: Zach Aller <[email protected]>
argoproj · May 7, 2024 · 16e67ce · 16e67ce
1 parent 1856c3c
commit 16e67ce
Show file tree

Hide file tree

Showing 21 changed files with 291 additions and 68 deletions.
diff --git a/.github/workflows/e2e.yaml b/.github/workflows/e2e.yaml
@@ -46,7 +46,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Set up Go
-        uses: actions/[email protected].0
+        uses: actions/[email protected].1
         with:
           go-version: '1.22'
       - uses: actions/checkout@v4
@@ -95,4 +95,4 @@ jobs:
         with:
           name: e2e-controller-k8s-${{ matrix.kubernetes-minor-version }}.log
           path: /tmp/e2e-controller.log
-        if: ${{ failure() }}
+        if: ${{ always() }}
diff --git a/.github/workflows/gh-pages.yaml b/.github/workflows/gh-pages.yaml
@@ -24,7 +24,7 @@ jobs:
         with:
           python-version: 3.x
       - name: Set up Go
-        uses: actions/[email protected].0
+        uses: actions/[email protected].1
         with:
           go-version: '1.22'
       - name: build

diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml
@@ -36,13 +36,13 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Set up Go
-        uses: actions/[email protected].0
+        uses: actions/[email protected].1
         with:
           go-version: ${{ env.GOLANG_VERSION }}
       - name: Checkout code
         uses: actions/checkout@v4
       - name: Run golangci-lint
-        uses: golangci/golangci-lint-action@v4
+        uses: golangci/golangci-lint-action@v5
         with:
           version: v1.57.2
           args: --timeout 6m
@@ -51,7 +51,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Set up Go
-        uses: actions/[email protected].0
+        uses: actions/[email protected].1
         with:
           go-version: ${{ env.GOLANG_VERSION }}
         id: go
@@ -90,7 +90,7 @@ jobs:
           path: coverage.out
 
       - name: Upload code coverage information to codecov.io
-        uses: codecov/[email protected].0
+        uses: codecov/[email protected].1
         with:
           file: coverage.out
 
@@ -103,7 +103,7 @@ jobs:
       - name: Checkout code
         uses: actions/checkout@v4
       - name: Setup Golang
-        uses: actions/[email protected].0
+        uses: actions/[email protected].1
         with:
           go-version: ${{ env.GOLANG_VERSION }}
       # k8s codegen generates files into GOPATH location instead of the GitHub git checkout location

diff --git a/.github/workflows/image-reuse.yaml b/.github/workflows/image-reuse.yaml
@@ -69,12 +69,12 @@ jobs:
         if: ${{ github.ref_type != 'tag'}}
 
       - name: Setup Golang
-        uses: actions/[email protected].0 # v3.5.0
+        uses: actions/[email protected].1 # v3.5.0
         with:
           go-version: ${{ inputs.go-version }}
 
       - name: Install cosign
-        uses: sigstore/cosign-installer@e1523de7571e31dbe865fd2e80c5c7c23ae71eb4 # v3.4.0
+        uses: sigstore/cosign-installer@59acb6260d9c0ba8f4a2f9d9b48431a222b68e20 # v3.5.0
         with:
           cosign-release: 'v2.2.0'
 

diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -51,7 +51,7 @@ jobs:
       id-token: write # for creating OIDC tokens for signing.
       packages: write # for uploading attestations. (https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/container/README.md#known-issues)
     # Must be refernced by a tag. https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/container/README.md#referencing-the-slsa-generator
-    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v1.10.0
+    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.0.0
     with:
       image: quay.io/argoproj/argo-rollouts
       digest: ${{ needs.controller-image.outputs.image-digest }}
@@ -67,7 +67,7 @@ jobs:
       id-token: write # for creating OIDC tokens for signing.
       packages: write # for uploading attestations. (https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/container/README.md#known-issues)
     # Must be refernced by a tag. https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/container/README.md#referencing-the-slsa-generator
-    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v1.10.0
+    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.0.0
     with:
       image: quay.io/argoproj/kubectl-argo-rollouts
       digest: ${{ needs.plugin-image.outputs.image-digest }}
@@ -90,7 +90,7 @@ jobs:
           token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Setup Golang
-        uses: actions/[email protected].0 # v4.0.1
+        uses: actions/[email protected].1 # v4.0.1
         with:
           go-version: ${{ env.GOLANG_VERSION }}
 
@@ -139,7 +139,7 @@ jobs:
       id-token: write # Needed for provenance signing and ID
       contents: write #  Needed for release uploads
     # Must be refernced by a tag. https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/container/README.md#referencing-the-slsa-generator
-    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.10.0
+    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.0.0
     with:
       base64-subjects: '${{ needs.release-artifacts.outputs.hashes }}'
       provenance-name: 'argo-rollouts.intoto.jsonl'
@@ -163,12 +163,12 @@ jobs:
           token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Setup Golang
-        uses: actions/[email protected].0 # v4.0.0
+        uses: actions/[email protected].1 # v4.0.0
         with:
           go-version: ${{ env.GOLANG_VERSION }}
 
       - name: Install cosign
-        uses: sigstore/cosign-installer@e1523de7571e31dbe865fd2e80c5c7c23ae71eb4 # v3.4.0
+        uses: sigstore/cosign-installer@59acb6260d9c0ba8f4a2f9d9b48431a222b68e20 # v3.5.0
         with:
           cosign-release: 'v2.2.0'
 

diff --git a/Makefile b/Makefile
@@ -22,7 +22,7 @@ DEV_IMAGE ?= false
 E2E_INSTANCE_ID ?= argo-rollouts-e2e
 E2E_TEST_OPTIONS ?=
 E2E_PARALLEL ?= 1
-E2E_WAIT_TIMEOUT ?= 120
+E2E_WAIT_TIMEOUT ?= 90
 GOPATH ?= $(shell go env GOPATH)
 
 # Global toolchain configuration
@@ -239,7 +239,7 @@ start-e2e: ## start e2e test environment
 
 .PHONY: test-e2e
 test-e2e: install-devtools-local
-	${DIST_DIR}/gotestsum --rerun-fails-report=rerunreport.txt --junitfile=junit.xml --format=testname --packages="./test/e2e" --rerun-fails=5 -- -timeout 60m -count 1 --tags e2e -p ${E2E_PARALLEL} -parallel ${E2E_PARALLEL} -v --short ./test/e2e ${E2E_TEST_OPTIONS}
+	${DIST_DIR}/gotestsum --rerun-fails-report=rerunreport.txt --junitfile=junit.xml --format=testname --packages="./test/e2e" --rerun-fails=5 -- -timeout 90m -count 1 --tags e2e -p ${E2E_PARALLEL} -parallel ${E2E_PARALLEL} -v --short ./test/e2e ${E2E_TEST_OPTIONS}
 
 .PHONY: test-unit
  test-unit: install-devtools-local ## run unit tests

diff --git a/docs/concepts.md b/docs/concepts.md
@@ -46,4 +46,33 @@ A Canary deployment exposes a subset of users to the new version of the applicat
 [![How Canary deployments work](concepts-assets/canary-deployments.png)](concepts-assets/canary-deployments.png)
 
 The picture above shows a canary with two stages (10% and 33% of traffic goes to new version) but this is just an example. With Argo Rollouts you can define the exact number of stages
-and percentages of traffic according to your use case.
+and percentages of traffic according to your use case.
+
+## Which strategy to choose
+
+In general Blue/Green is the easier strategy to start with, but also the more limited. We recommend you start with Blue/Green deployments first and as you gain confidence for your metrics and applications switch to Canaries.
+
+You also need to examine if your application can handle canaries or not.
+
+* Blue/Green always works because only one application is active at a time. Not all applications can have different versions running in parallel at the same time (which is what canaries are doing). This can be a showstopper for adopting canary deployments especially for legacy applications.
+* Blue/Green is simpler because you can get their full value WITHOUT a traffic manager. While canaries can also work without a traffic manager, most of their advanced features assume a fine-grained way to control traffic. If you don't have a traffic manager, then you can easily get the full value
+of blue/green deployments but only the basic capabilities of canaries.
+* Blue/Green also works with services that use queues and databases (workers that fetch tasks). Argo Rollouts doesn't control traffic flow for
+connections it doesn't understand (i.e. binary/queue channels).
+
+Here is a summary table for the possible approaches.
+
+|                           |         Blue/Green        |       Basic Canary         | Canary with Traffic manager    |
+|--------------------------:|:-------------------------:|:--------------------------:| :-----------------------------:|
+|       Adoption Complexity |           Low             |       Medium               |        High                    |
+|        Flexibility        |           Low             |        High                |        Maximum                 |
+|    Needs traffic provider |                 No        |         No                 |           Yes                  | 
+|  Works with queue workers |                Yes        |         No                 |            No                  |
+|  Works with shared/locked resources | Yes             |         No                 |            No                  |
+|            Traffic switch |     All or nothing        |  Gradual percentage        |      Gradual percentage        |
+|           Traffic control |     0% or 100%            |  coarse grained            |     fine grained               |
+|       Traffic depends on  |     deployment state      |  number of canary pods     |     Any split option is possible                |
+| Advanced routing scenarios |         No               |       No                   |         Yes                    |
+|      Failure Blast Radius | Massive impact            |  Low impact                       | Low impact              |
+
+Note that that traffic manager can be any compatible Service Mesh or Ingress Controller or Gateway API implementation (via a plugin).
diff --git a/experiments/replicaset.go b/experiments/replicaset.go
@@ -6,6 +6,8 @@ import (
 	"fmt"
 	"time"
 
+	"github.com/argoproj/argo-rollouts/utils/diff"
+
 	log "github.com/sirupsen/logrus"
 	appsv1 "k8s.io/api/apps/v1"
 	"k8s.io/apimachinery/pkg/api/errors"
@@ -287,16 +289,39 @@ func (ec *experimentContext) scaleReplicaSet(rs *appsv1.ReplicaSet, newScale int
 	sizeNeedsUpdate := oldScale != newScale
 	scaled := false
 	var err error
+	var updatedRS *appsv1.ReplicaSet
 	if sizeNeedsUpdate {
 		rsCopy := rs.DeepCopy()
 		*(rsCopy.Spec.Replicas) = newScale
-		rs, err = ec.kubeclientset.AppsV1().ReplicaSets(rsCopy.Namespace).Update(ctx, rsCopy, metav1.UpdateOptions{})
+
+		updatedRS, err = ec.kubeclientset.AppsV1().ReplicaSets(rsCopy.Namespace).Update(ctx, rsCopy, metav1.UpdateOptions{})
+		if err != nil {
+			if errors.IsConflict(err) {
+				ec.log.Infof("Conflict when updating replicaset %s, falling back to patch", rs.Name)
+
+				patchRS := appsv1.ReplicaSet{}
+				patchRS.Spec.Replicas = rsCopy.Spec.Replicas
+
+				patch, changed, err := diff.CreateTwoWayMergePatch(appsv1.ReplicaSet{}, patchRS, appsv1.ReplicaSet{})
+				if err != nil {
+					return scaled, nil, err
+				}
+
+				if changed {
+					ec.log.Infof("Patching expirment replicaset with patch: %s", string(patch))
+					updatedRS, err = ec.kubeclientset.AppsV1().ReplicaSets(rs.Namespace).Patch(ctx, rs.Name, patchtypes.StrategicMergePatchType, patch, metav1.PatchOptions{})
+					if err != nil {
+						return scaled, nil, err
+					}
+				}
+			}
+		}
 		if err == nil && sizeNeedsUpdate {
 			scaled = true
 			ec.recorder.Eventf(ec.ex, record.EventOptions{EventReason: conditions.ScalingReplicaSetReason}, "Scaled %s ReplicaSet %s from %d to %d", scalingOperation, rs.Name, oldScale, newScale)
 		}
 	}
-	return scaled, rs, err
+	return scaled, updatedRS, err
 }
 
 func newReplicaSetAnnotations(experimentName, templateName string) map[string]string {

diff --git a/go.mod b/go.mod
@@ -10,7 +10,7 @@ require (
 	github.com/argoproj/pkg v0.13.6
 	github.com/aws/aws-sdk-go-v2 v1.26.1
 	github.com/aws/aws-sdk-go-v2/config v1.27.11
-	github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.37.0
+	github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.38.0
 	github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.30.5
 	github.com/aws/smithy-go v1.20.2
 	github.com/blang/semver v3.5.1+incompatible
@@ -42,7 +42,7 @@ require (
 	golang.org/x/oauth2 v0.19.0
 	google.golang.org/genproto/googleapis/api v0.0.0-20240227224415-6ceb2ff114de
 	google.golang.org/grpc v1.63.2
-	google.golang.org/protobuf v1.33.0
+	google.golang.org/protobuf v1.34.0
 	gopkg.in/yaml.v2 v2.4.0
 	k8s.io/api v0.29.3
 	k8s.io/apiextensions-apiserver v0.29.3

diff --git a/go.sum b/go.sum
@@ -112,8 +112,8 @@ github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.5 h1:PG1F3OD1szkuQPzDw3C
 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.5/go.mod h1:jU1li6RFryMz+so64PpKtudI+QzbKoIEivqdf6LNpOc=
 github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 h1:hT8rVHwugYE2lEfdFE0QWVo81lF7jMrYJVDWI+f+VxU=
 github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0/go.mod h1:8tu/lYfQfFe6IGnaOdrpVgEL2IrrDOf6/m9RQum4NkY=
-github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.37.0 h1:sGGUnU/pUSzjrcCvQgN2pEc3aTQILyK2rRsWVY5CSt0=
-github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.37.0/go.mod h1:U12sr6Lt14X96f16t+rR52+2BdqtydwN7DjEEHRMjO0=
+github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.38.0 h1:vAfGwYFCcPDS9Bg7ckfMBer6olJLOHsOAVoKWpPIirs=
+github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.38.0/go.mod h1:U12sr6Lt14X96f16t+rR52+2BdqtydwN7DjEEHRMjO0=
 github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.30.5 h1:/x2u/TOx+n17U+gz98TOw1HKJom0EOqrhL4SjrHr0cQ=
 github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.30.5/go.mod h1:e1McVqsud0JOERidvppLEHnuCdh/X6MRyL5L0LseAUk=
 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.2 h1:Ji0DY1xUsUr3I8cHps0G+XM3WWU16lP6yG8qu1GAZAs=
@@ -1147,8 +1147,8 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba
 google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
 google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
 google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
-google.golang.org/protobuf v1.33.0 h1:uNO2rsAINq/JlFpSdYEKIZ0uKD/R9cpdv0T+yoGwGmI=
-google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
+google.golang.org/protobuf v1.34.0 h1:Qo/qEd2RZPCf2nKuorzksSknv0d3ERwp1vFG38gSmH4=
+google.golang.org/protobuf v1.34.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
 gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc h1:2gGKlE2+asNV9m7xrywl36YYNnBG5ZQ0r/BOOxqPpmk=
 gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc/go.mod h1:m7x9LTH6d71AHyAX77c9yqWCCa3UKHcVEj9y7hAtKDk=

diff --git a/hack/gen-crd-spec/main.go b/hack/gen-crd-spec/main.go
@@ -96,11 +96,12 @@ func NewCustomResourceDefinition() []*extensionsobj.CustomResourceDefinition {
 	// clean up stuff left by controller-gen
 	deleteFile("config/webhook/manifests.yaml")
 	deleteFile("config/webhook")
-	deleteFile("config/argoproj.io_analysisruns.yaml")
-	deleteFile("config/argoproj.io_analysistemplates.yaml")
-	deleteFile("config/argoproj.io_clusteranalysistemplates.yaml")
-	deleteFile("config/argoproj.io_experiments.yaml")
-	deleteFile("config/argoproj.io_rollouts.yaml")
+	deleteFile("config/crd/argoproj.io_analysisruns.yaml")
+	deleteFile("config/crd/argoproj.io_analysistemplates.yaml")
+	deleteFile("config/crd/argoproj.io_clusteranalysistemplates.yaml")
+	deleteFile("config/crd/argoproj.io_experiments.yaml")
+	deleteFile("config/crd/argoproj.io_rollouts.yaml")
+	deleteFile("config/crd")
 	deleteFile("config")
 
 	crds := []*extensionsobj.CustomResourceDefinition{}

diff --git a/rollout/canary.go b/rollout/canary.go
@@ -112,7 +112,7 @@ func (c *rolloutContext) reconcileCanaryStableReplicaSet() (bool, error) {
 	}
 	scaled, _, err := c.scaleReplicaSetAndRecordEvent(c.stableRS, desiredStableRSReplicaCount)
 	if err != nil {
-		return scaled, fmt.Errorf("failed to scaleReplicaSetAndRecordEvent in reconcileCanaryStableReplicaSet:L %w", err)
+		return scaled, fmt.Errorf("failed to scaleReplicaSetAndRecordEvent in reconcileCanaryStableReplicaSet: %w", err)
 	}
 	return scaled, err
 }