V1.3.0 rallyhealth pr (#6)

* feat: add support for influxdb as a metrics provider (argoproj#1839) * feat: add InfluxDB metric provider implementation Signed-off-by: Jayme Bird <[email protected]> * feat: add influx to metric provider util count, add missing graphite Signed-off-by: Jayme Bird <[email protected]> * feat: add protobuf definitions for InfluxdbMetric Signed-off-by: Jayme Bird <[email protected]> * feat: add docs for the Influxdb metrics provider Signed-off-by: Jayme Bird <[email protected]> * feat: add tests for influxdb metrics provider Signed-off-by: Jayme Bird <[email protected]> * feat: add go mod replace for version of moq due to security issue Signed-off-by: Jayme Bird <[email protected]> * feat: update error message to distinguish failed cases - review feedback Signed-off-by: Jayme Bird <[email protected]> * feat: update tests to simplify mocking of QueryTableResult An upstream change to the influx go client makes it easier to mock QueryTableResult in tests. This removes a lot of boilerplate duplication in the tests. Signed-off-by: Jayme Bird <[email protected]> * feat: re-run codegen Signed-off-by: Jayme Bird <[email protected]> * feat: Adds support for Istio traffic mirroring (argoproj#2074) * feat: Add support for istio traffic mirroring steps Signed-off-by: zachaller <[email protected]> * bump e2e timeout Signed-off-by: zachaller <[email protected]> * Remove unintended change Signed-off-by: zachaller <[email protected]> * Cleaner events Signed-off-by: zachaller <[email protected]> * Add docs for mirror and fix up header routing Signed-off-by: zachaller <[email protected]> * small doc changes Signed-off-by: zachaller <[email protected]> * Add example Signed-off-by: zachaller <[email protected]> * Fix types and change example Signed-off-by: zachaller <[email protected]> * Remove unused type Signed-off-by: zachaller <[email protected]> * Docs change Signed-off-by: zachaller <[email protected]> * Fix PR comments Signed-off-by: zachaller <[email protected]> * Check for >= Signed-off-by: zachaller <[email protected]> * Remove unused function Signed-off-by: zachaller <[email protected]> * codegen Signed-off-by: zachaller <[email protected]> * Add better error logging Signed-off-by: zachaller <[email protected]> * a few more error msgs Signed-off-by: zachaller <[email protected]> * fix comments Signed-off-by: zachaller <[email protected]> * typo Signed-off-by: zachaller <[email protected]> * fix logic error Signed-off-by: zachaller <[email protected]> * redo header logic Signed-off-by: zachaller <[email protected]> * cleanup Signed-off-by: zachaller <[email protected]> * Fix issue with keeping route weight updated when no routes are defined in spec.strategy.canary.trafficRouting.istio.virtualService.routes requiring only a single route Signed-off-by: zachaller <[email protected]> * cleanup routes when promiting with last step being a pause Signed-off-by: zachaller <[email protected]> * fix bug with promote and last step is a pause Signed-off-by: zachaller <[email protected]> * Fix tests Signed-off-by: zachaller <[email protected]> * fix typo Signed-off-by: zachaller <[email protected]> * chore: improve openapi schema (argoproj#2081) * feat: include all CRDs to the OpenAPI schema Signed-off-by: Mário Bezerra <[email protected]> * feat: use kube-openapi naming convention in the OpenAPI schema Signed-off-by: Mário Bezerra <[email protected]> * chore: generate OpenAPI schema Signed-off-by: Mário Bezerra <[email protected]> * feat(grafana): Allow selecting datasource for grafana dashboard (argoproj#1988) Signed-off-by: Jesse Antoszyk <[email protected]> * feat: support /rollouts/:namespace?q=... and /rollout/:namespace/:name (argoproj#1902) Signed-off-by: Simon Ninon <[email protected]> * chore: Add e2e and unit test comment reports (argoproj#2123) Signed-off-by: zachaller <[email protected]> * chore: upgrade deps (argoproj#2136) Signed-off-by: zachaller <[email protected]> * fixes argoproj#2141 Added list and watch to clusterrole (argoproj#2145) Signed-off-by: Martin Adler <[email protected]> * fix: Update ro.Status.ALB when first creating rollout object (argoproj#1986) * fix: make sure we update rollout.Status.ALB when first create Rollout object This fixes the case when we first create a rollout the status.ALB field does not get updated with valid information form the elb. Signed-off-by: zachaller <[email protected]> * Refactor check for if we should verify alb. This creates a new rolloututil function ShouldVerifyWeight that verifies that we should call aws because the rollout is in the middle of some update. The reason I feel this should move into the ingress implmentation function VerifyWeight is becuase in the future other ingresses might also need to verify weights and adding ingress specific status checks in trafficrouting.go like a leak details where it should be up to the ingress provider to determin if it should be careful of rate limiting do to say it being a cloud provider call. Signed-off-by: zachaller <[email protected]> * lint Signed-off-by: zachaller <[email protected]> * Fix tests Signed-off-by: zachaller <[email protected]> * Add new test for ShouldVerifyWeight Signed-off-by: zachaller <[email protected]> * Fix logic Signed-off-by: zachaller <[email protected]> * Add test for where we do not need to verify weight and its already set Signed-off-by: zachaller <[email protected]> * fix up review comments Signed-off-by: zachaller <[email protected]> * fix tests Signed-off-by: zachaller <[email protected]> * fix bad merge Signed-off-by: zachaller <[email protected]> * Fix test Signed-off-by: zachaller <[email protected]> * Fix test Signed-off-by: zachaller <[email protected]> * Clearer function name Signed-off-by: zachaller <[email protected]> * Clean up if logic Signed-off-by: zachaller <[email protected]> * fix: remove metrics when objects are removed from cluster to prevent build up (argoproj#2115) * fix: remove metrics on resource removal Signed-off-by: zachaller <[email protected]> * Add test Signed-off-by: zachaller <[email protected]> * More tests Signed-off-by: zachaller <[email protected]> * back off deps updates Signed-off-by: zachaller <[email protected]> * Fix dep change Signed-off-by: zachaller <[email protected]> * upgrade prom deps for new features Signed-off-by: zachaller <[email protected]> * fix deps Signed-off-by: zachaller <[email protected]> * chore: use controler-gen for cluster analysis template scope (argoproj#2148) Signed-off-by: zachaller <[email protected]> * fix(analysis): Fix Analysis Terminal Decision For Dry-Run Metrics (argoproj#2131) Signed-off-by: Rohit Agrawal <[email protected]> * docs: update release doc with brew formula details (argoproj#2165) Signed-off-by: Leonardo Luz Almeida <[email protected]> * fix(docs) Graphite metrics provider linked in docs sidebar. Fixes argoproj#2102. (argoproj#2094) * Graphite metrics provider linked in docs sidebar This fixes an issue wherein the Graphite metrics provider has no sidebar link in the docs hosted at https://argoproj.github.io/argo-rollouts/. As a bonus, this also removes various trailing whitespace from analysis docs. Signed-off-by: Mike Ball <[email protected]> * build trigger Signed-off-by: Mike Ball <[email protected]> * feat: emit rollout delete event (argoproj#1893) Signed-off-by: Hui Kang <[email protected]> Co-authored-by: Hui Kang <[email protected]> * chore: Upgrade golang (argoproj#2160) * Upgrade golang Signed-off-by: zachaller <[email protected]> * upgrade golang to 1.18 for e2e Signed-off-by: zachaller <[email protected]> * Fix deps Signed-off-by: zachaller <[email protected]> * Update build action Signed-off-by: zachaller <[email protected]> * Upgrade push action Signed-off-by: zachaller <[email protected]> * Update docker file Signed-off-by: zachaller <[email protected]> * Bump golang lint to match Signed-off-by: zachaller <[email protected]> * fix go.mod Signed-off-by: zachaller <[email protected]> * fix: Failed to process: Object 'Kind' is missing in Errors with rollouts notification (argoproj#2150) * fix: update rolloutobject with gvk before writing to rollout informer Signed-off-by: Ravi Hari <[email protected]> * fix: controller schema linting Signed-off-by: Ravi Hari <[email protected]> * docs: comment the details on the change Signed-off-by: Ravi Hari <[email protected]> * docs: comment the details on the change Signed-off-by: Ravi Hari <[email protected]> * fix: rootPath support so that it uses the embedded files system (argoproj#2198) * fix: rootPath support so that it uses the embeded files system Signed-off-by: zachaller <[email protected]> * Catch edge cases and make sure we always server index.html on not found Signed-off-by: zachaller <[email protected]> * turn path.Clean into var Signed-off-by: zachaller <[email protected]> Signed-off-by: zachaller <[email protected]> * fix: change completed condition so it only triggers on pod hash changes also adds an event for when it does changes. (argoproj#2203) * feat: add healthy event/condition and fix completed event/condition Signed-off-by: zachaller <[email protected]> * fix unit tests Signed-off-by: zachaller <[email protected]> * rename vars to make more sense and remove healthy event becase it will never be consistent Signed-off-by: zachaller <[email protected]> * fix unit tests Signed-off-by: zachaller <[email protected]> * possible fix for e2e Signed-off-by: zachaller <[email protected]> * fix e2e Signed-off-by: zachaller <[email protected]> * unit test for complete function Signed-off-by: zachaller <[email protected]> * small cleanup and changes to not check generation Signed-off-by: zachaller <[email protected]> * fix unit test and proper behavior Signed-off-by: zachaller <[email protected]> * Fix e2e Signed-off-by: zachaller <[email protected]> * rename and fix one unit test Signed-off-by: zachaller <[email protected]> * fix unit tests Signed-off-by: zachaller <[email protected]> * fix unit test Signed-off-by: zachaller <[email protected]> * add seperate test for TestRolloutComplete Signed-off-by: zachaller <[email protected]> * renames Signed-off-by: zachaller <[email protected]> * fix e2e Signed-off-by: zachaller <[email protected]> * Set Completed to false Signed-off-by: zachaller <[email protected]> * Add event Signed-off-by: zachaller <[email protected]> * fix e2e Signed-off-by: zachaller <[email protected]> * refactor Signed-off-by: zachaller <[email protected]> * Fix all but one unit test Signed-off-by: zachaller <[email protected]> * fix last unit test Signed-off-by: zachaller <[email protected]> * lint Signed-off-by: zachaller <[email protected]> * cleanup Signed-off-by: zachaller <[email protected]> * Rename Signed-off-by: zachaller <[email protected]> * More renames Signed-off-by: zachaller <[email protected]> * small comment change Signed-off-by: zachaller <[email protected]> Signed-off-by: zachaller <[email protected]> * feat: Add support for spec.ingressClassName (argoproj#2178) This change adds support for `spec.ingressClassName` while still supporting `kubernetes.io/ingress.class` annotation for backwards compatibility. Fixes argoproj#1277 Signed-off-by: Siavash Safi <[email protected]> Signed-off-by: Siavash Safi <[email protected]> * fix: enable notifications without when condition (argoproj#2231) * fix: enable notifications without when condition Signed-off-by: Ravi Hari <[email protected]> * fix: use trigger action item from the list Signed-off-by: Ravi Hari <[email protected]> * fix: add emptycondition logic to make notifications work with/without conditions Signed-off-by: Ravi Hari <[email protected]> * fix: linting Signed-off-by: Ravi Hari <[email protected]> Signed-off-by: Ravi Hari <[email protected]> * Use standard cli format for dashboard root path (argoproj#2244) Signed-off-by: zachaller <[email protected]> Signed-off-by: zachaller <[email protected]> * merge conflict broke tests Signed-off-by: Travis Perdue <[email protected]> Signed-off-by: Jayme Bird <[email protected]> Signed-off-by: zachaller <[email protected]> Signed-off-by: Mário Bezerra <[email protected]> Signed-off-by: Jesse Antoszyk <[email protected]> Signed-off-by: Simon Ninon <[email protected]> Signed-off-by: Martin Adler <[email protected]> Signed-off-by: Rohit Agrawal <[email protected]> Signed-off-by: Leonardo Luz Almeida <[email protected]> Signed-off-by: Mike Ball <[email protected]> Signed-off-by: Ravi Hari <[email protected]> Signed-off-by: Siavash Safi <[email protected]> Signed-off-by: Travis Perdue <[email protected]> Co-authored-by: jaymebrd <[email protected]> Co-authored-by: Zach Aller <[email protected]> Co-authored-by: Mário Bezerra <[email protected]> Co-authored-by: Jesse Antoszyk <[email protected]> Co-authored-by: Simon Ninon <[email protected]> Co-authored-by: Martin Adler <[email protected]> Co-authored-by: Rohit Agrawal <[email protected]> Co-authored-by: Leonardo Luz Almeida <[email protected]> Co-authored-by: Mike Ball <[email protected]> Co-authored-by: cskh <[email protected]> Co-authored-by: Hui Kang <[email protected]> Co-authored-by: RaviHari <[email protected]> Co-authored-by: Siavash Safi <[email protected]>
rallyhealth · Oct 18, 2022 · a1af660 · a1af660
1 parent 93d05cd
commit a1af660
Show file tree

Hide file tree

Showing 132 changed files with 24,360 additions and 1,871 deletions.
diff --git a/.github/workflows/docker-publish.yml b/.github/workflows/docker-publish.yml
@@ -21,7 +21,7 @@ jobs:
         uses: docker/setup-qemu-action@v1
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v1
+        uses: docker/setup-buildx-action@v2
         with:
           config-inline: |
             [worker.oci]
@@ -79,14 +79,14 @@ jobs:
           echo "::set-output name=platform-matrix::$PLATFORM_MATRIX"
 
       - name: Build and push (controller-image)
-        uses: docker/build-push-action@v2
+        uses: docker/build-push-action@v3
         with:
           platforms: ${{ steps.platform-matrix.outputs.platform-matrix }}
           push: ${{ github.event_name != 'pull_request' }}
           tags: ${{ steps.controller-meta.outputs.tags }}
 
       - name: Build and push (plugin-image)
-        uses: docker/build-push-action@v2
+        uses: docker/build-push-action@v3
         with:
           target: kubectl-argo-rollouts
           platforms: ${{ steps.platform-matrix.outputs.platform-matrix }}

diff --git a/.github/workflows/e2e-test-results.yml b/.github/workflows/e2e-test-results.yml
@@ -0,0 +1,43 @@
+name: Test Results
+
+on:
+  workflow_run:
+    workflows: ["E2E Tests", "Go"]
+    types:
+      - completed
+permissions: {}
+
+jobs:
+  test-results:
+    name: "${{ github.event.workflow.name }} Test Results"
+    runs-on: ubuntu-latest
+    if: github.event.workflow_run.conclusion != 'skipped'
+    permissions:
+      checks: write
+      pull-requests: write
+      actions: read
+    steps:
+      - name: Download and Extract Artifacts
+        env:
+          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
+        run: |
+          mkdir -p artifacts && cd artifacts
+          artifacts_url=${{ github.event.workflow_run.artifacts_url }}
+          gh api "$artifacts_url" -q '.artifacts[] | [.name, .archive_download_url] | @tsv' | while read artifact
+          do
+            IFS=$'\t' read name url <<< "$artifact"
+            gh api $url > "$name.zip"
+            unzip -d "$name" "$name.zip"
+          done
+
+      - name: Publish Test Results
+        uses: EnricoMi/publish-unit-test-result-action@v1
+        with:
+          check_name: "${{ github.event.workflow.name }} Published Test Results"
+          commit: ${{ github.event.workflow_run.head_sha }}
+          event_file: artifacts/Event File/event.json
+          event_name: ${{ github.event.workflow_run.event }}
+          files: "artifacts/**/*.xml"
+          compare_to_earlier_commit: false
+          test_changes_limit: 0
+          fail_on: "errors"
diff --git a/.github/workflows/e2e.yaml b/.github/workflows/e2e.yaml
@@ -21,14 +21,23 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
+  event_file:
+    name: "Event File"
+    runs-on: ubuntu-latest
+    steps:
+      - name: Upload
+        uses: actions/upload-artifact@v2
+        with:
+          name: Event File
+          path: ${{ github.event_path }}
   test-e2e:
     name: Run end-to-end tests
     runs-on: ubuntu-latest
     steps:
       - name: Set up Go
         uses: actions/setup-go@v2
         with:
-          go-version: 1.17
+          go-version: 1.18
       - uses: actions/checkout@v2
       - name: Setup k3s
         run: |
@@ -58,6 +67,16 @@ jobs:
       - name: Run e2e tests
         run: make test-e2e
         if: ${{ !(github.event_name == 'workflow_dispatch' && github.event.inputs.debug_enabled) }}
+      - name: Output Rerun Overview
+        run: |
+          [[ -f rerunreport.txt ]] && cat rerunreport.txt || echo "No rerun report found"
+      - name: Upload E2E Test Results
+        if: always()
+        uses: actions/upload-artifact@v2
+        with:
+          name: E2E Test Results
+          path: |
+            junit.xml
       - name: Upload e2e-controller logs
         uses: actions/upload-artifact@v2
         with:

diff --git a/.github/workflows/gh-pages.yaml b/.github/workflows/gh-pages.yaml
@@ -21,7 +21,7 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@v2
         with:
-          go-version: 1.17
+          go-version: 1.18
       - name: build
         run: |
           pip install mkdocs mkdocs_material

diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml
@@ -9,9 +9,18 @@ on:
       - "master"
 env:
   # Golang version to use across CI steps
-  GOLANG_VERSION: '1.17'
+  GOLANG_VERSION: '1.18'
 
 jobs:
+  event_file:
+    name: "Event File"
+    runs-on: ubuntu-latest
+    steps:
+      - name: Upload
+        uses: actions/upload-artifact@v2
+        with:
+          name: Event File
+          path: ${{ github.event_path }}
   lint-go:
     name: Lint Go code
     runs-on: ubuntu-latest
@@ -25,7 +34,7 @@ jobs:
       - name: Run golangci-lint
         uses: golangci/golangci-lint-action@v3
         with:
-          version: v1.45.2
+          version: v1.47.2
           args: --timeout 5m
   build:
     name: Build
@@ -54,7 +63,15 @@ jobs:
         run: make controller plugin
 
       - name: Test
-        run: go test -failfast -covermode=count -coverprofile=coverage.out ./...
+        run: make test-unit
+
+      - name: Upload Unit Test Results
+        if: always()
+        uses: actions/upload-artifact@v2
+        with:
+          name: Unit Test Results
+          path: |
+            junit.xml
 
       - name: Generate code coverage artifacts
         uses: actions/upload-artifact@v2

diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -106,7 +106,7 @@ jobs:
       - name: Setup Golang
         uses: actions/setup-go@v2
         with:
-          go-version: 1.17
+          go-version: 1.18
 
       - name: Generate release artifacts
         run: |

diff --git a/Dockerfile b/Dockerfile
@@ -3,7 +3,7 @@
 # Initial stage which pulls prepares build dependencies and CLI tooling we need for our final image
 # Also used as the image in CI jobs so needs all dependencies
 ####################################################################################################
-FROM --platform=$BUILDPLATFORM golang:1.17 as builder
+FROM --platform=$BUILDPLATFORM golang:1.18 as builder
 
 RUN apt-get update && apt-get install -y \
     wget \
@@ -12,7 +12,7 @@ RUN apt-get update && apt-get install -y \
     rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 
 # Install golangci-lint
-RUN curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.44.0 && \
+RUN curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.47.2 && \
     golangci-lint linters
 
 COPY .golangci.yml ${GOPATH}/src/dummy/.golangci.yml
@@ -40,7 +40,7 @@ RUN NODE_ENV='production' yarn build
 ####################################################################################################
 # Rollout Controller Build stage which performs the actual build of argo-rollouts binaries
 ####################################################################################################
-FROM --platform=$BUILDPLATFORM golang:1.17 as argo-rollouts-build
+FROM --platform=$BUILDPLATFORM golang:1.18 as argo-rollouts-build
 
 WORKDIR /go/src/github.com/argoproj/argo-rollouts
 

diff --git a/Makefile b/Makefile
@@ -21,8 +21,8 @@ DEV_IMAGE=false
 # E2E variables
 E2E_INSTANCE_ID ?= argo-rollouts-e2e
 E2E_TEST_OPTIONS ?= 
-E2E_PARALLEL ?= 4
-E2E_WAIT_TIMEOUT ?= 90
+E2E_PARALLEL ?= 1
+E2E_WAIT_TIMEOUT ?= 120
 
 override LDFLAGS += \
   -X ${PACKAGE}/utils/version.version=${VERSION} \
@@ -79,9 +79,13 @@ install-go-tools-local: go-mod-vendor
 install-protoc-local:
 	./hack/installers/install-protoc.sh
 
+.PHONY: install-devtools-local
+install-devtools-local:
+	./hack/installers/install-dev-tools.sh
+
 # Installs all tools required to build and test locally
 .PHONY: install-tools-local
-install-tools-local: install-go-tools-local install-protoc-local
+install-tools-local: install-go-tools-local install-protoc-local install-devtools-local
 
 TYPES := $(shell find pkg/apis/rollouts/v1alpha1 -type f -name '*.go' -not -name openapi_generated.go -not -name '*generated*' -not -name '*test.go')
 APIMACHINERY_PKGS=k8s.io/apimachinery/pkg/util/intstr,+k8s.io/apimachinery/pkg/api/resource,+k8s.io/apimachinery/pkg/runtime/schema,+k8s.io/apimachinery/pkg/runtime,k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/api/core/v1,k8s.io/api/batch/v1
@@ -212,8 +216,13 @@ start-e2e:
 	go run ./cmd/rollouts-controller/main.go --instance-id ${E2E_INSTANCE_ID} --loglevel debug --kloglevel 6
 
 .PHONY: test-e2e
-test-e2e:
-	go test -timeout 30m -v -count 1 --tags e2e -p ${E2E_PARALLEL} --short ./test/e2e ${E2E_TEST_OPTIONS}
+test-e2e: install-devtools-local
+	${DIST_DIR}/gotestsum --rerun-fails-report=rerunreport.txt --junitfile=junit.xml --format=testname --packages="./test/e2e" --rerun-fails=5 -- -timeout 60m -count 1 --tags e2e -p ${E2E_PARALLEL} -parallel ${E2E_PARALLEL} -v --short ./test/e2e ${E2E_TEST_OPTIONS}
+
+.PHONY: test-unit
+ test-unit: install-devtools-local
+	${DIST_DIR}/gotestsum --junitfile=junit.xml --format=testname --packages="./..." -- -covermode=count -coverprofile=coverage.out ./...
+
 
 .PHONY: coverage
 coverage: test

diff --git a/README.md b/README.md
@@ -39,7 +39,7 @@ For these reasons, in large scale high-volume production environments, a rolling
 * Customizable metric queries and analysis of business KPIs
 * Ingress controller integration: NGINX, ALB
 * Service Mesh integration: Istio, Linkerd, SMI
-* Metric provider integration: Prometheus, Wavefront, Kayenta, Web, Kubernetes Jobs, Datadog, New Relic
+* Metric provider integration: Prometheus, Wavefront, Kayenta, Web, Kubernetes Jobs, Datadog, New Relic, InfluxDB
 
 ## Documentation
 To learn more about Argo Rollouts go to the [complete documentation](https://argoproj.github.io/argo-rollouts/).

diff --git a/USERS.md b/USERS.md
@@ -19,6 +19,7 @@ Organizations below are **officially** using Argo Rollouts. Please send a PR wit
 1. [New Relic](https://newrelic.com/)
 1. [Nitro](https://www.gonitro.com)
 1. [Nozzle](https://nozzle.io)
+1. [PagerDuty](https://www.pagerduty.com/)
 1. [PayPal](https://www.paypal.com/)
 1. [PayPay](https://paypay.ne.jp/)
 1. [Quipper](https://www.quipper.com/)

diff --git a/analysis/analysis_test.go b/analysis/analysis_test.go
@@ -1500,7 +1500,7 @@ func TestAssessRunStatusErrorMessageAnalysisPhaseFail(t *testing.T) {
 
 func TestAssessRunStatusErrorMessageAnalysisPhaseFailInDryRunMode(t *testing.T) {
 	status, message, dryRunSummary := StartAssessRunStatusErrorMessageAnalysisPhaseFail(t, true)
-	assert.Equal(t, v1alpha1.AnalysisPhaseSuccessful, status)
+	assert.Equal(t, v1alpha1.AnalysisPhaseRunning, status)
 	assert.Equal(t, "", message)
 	expectedDryRunSummary := v1alpha1.RunSummary{
 		Count:        2,
@@ -1545,7 +1545,7 @@ func TestAssessRunStatusErrorMessageFromProvider(t *testing.T) {
 func TestAssessRunStatusErrorMessageFromProviderInDryRunMode(t *testing.T) {
 	providerMessage := "Provider Error"
 	status, message, dryRunSummary := StartAssessRunStatusErrorMessageFromProvider(t, providerMessage, true)
-	assert.Equal(t, v1alpha1.AnalysisPhaseSuccessful, status)
+	assert.Equal(t, v1alpha1.AnalysisPhaseRunning, status)
 	assert.Equal(t, "", message)
 	expectedDryRunSummary := v1alpha1.RunSummary{
 		Count:        2,
@@ -1587,7 +1587,7 @@ func TestAssessRunStatusMultipleFailures(t *testing.T) {
 
 func TestAssessRunStatusMultipleFailuresInDryRunMode(t *testing.T) {
 	status, message, dryRunSummary := StartAssessRunStatusMultipleFailures(t, true)
-	assert.Equal(t, v1alpha1.AnalysisPhaseSuccessful, status)
+	assert.Equal(t, v1alpha1.AnalysisPhaseRunning, status)
 	assert.Equal(t, "", message)
 	expectedDryRunSummary := v1alpha1.RunSummary{
 		Count:        2,
@@ -1623,7 +1623,7 @@ func TestAssessRunStatusWorstMessageInReconcileAnalysisRun(t *testing.T) {
 
 func TestAssessRunStatusWorstMessageInReconcileAnalysisRunInDryRunMode(t *testing.T) {
 	newRun := StartAssessRunStatusWorstMessageInReconcileAnalysisRun(t, true)
-	assert.Equal(t, v1alpha1.AnalysisPhaseSuccessful, newRun.Status.Phase)
+	assert.Equal(t, v1alpha1.AnalysisPhaseRunning, newRun.Status.Phase)
 	assert.Equal(t, "", newRun.Status.Message)
 	expectedDryRunSummary := v1alpha1.RunSummary{
 		Count:        2,

diff --git a/analysis/controller.go b/analysis/controller.go
@@ -3,6 +3,8 @@ package analysis
 import (
 	"time"
 
+	unstructuredutil "github.com/argoproj/argo-rollouts/utils/unstructured"
+
 	log "github.com/sirupsen/logrus"
 	batchv1 "k8s.io/api/batch/v1"
 	k8serrors "k8s.io/apimachinery/pkg/api/errors"
@@ -117,7 +119,14 @@ func NewController(cfg ControllerConfig) *Controller {
 		UpdateFunc: func(old, new interface{}) {
 			controller.enqueueAnalysis(new)
 		},
-		DeleteFunc: controller.enqueueAnalysis,
+		DeleteFunc: func(obj interface{}) {
+			controller.enqueueAnalysis(obj)
+			if ar := unstructuredutil.ObjectToAnalysisRun(obj); ar != nil {
+				logCtx := logutil.WithAnalysisRun(ar)
+				logCtx.Info("analysis run deleted")
+				controller.metricsServer.Remove(ar.Namespace, ar.Name, logutil.AnalysisRunKey)
+			}
+		},
 	})
 	return controller
 }

diff --git a/controller/metrics/analysis_test.go b/controller/metrics/analysis_test.go
@@ -5,6 +5,8 @@ import (
 	"testing"
 	"time"
 
+	"github.com/stretchr/testify/assert"
+
 	"github.com/ghodss/yaml"
 	"github.com/prometheus/client_golang/prometheus"
 	"github.com/prometheus/client_golang/prometheus/promhttp"
@@ -162,7 +164,7 @@ func testAnalysisRunDescribe(t *testing.T, fakeAnalysisRun string, expectedRespo
 	registry.MustRegister(NewAnalysisRunCollector(serverCfg.AnalysisRunLister, serverCfg.AnalysisTemplateLister, serverCfg.ClusterAnalysisTemplateLister))
 	mux := http.NewServeMux()
 	mux.Handle(MetricsPath, promhttp.HandlerFor(registry, promhttp.HandlerOpts{}))
-	testHttpResponse(t, mux, expectedResponse)
+	testHttpResponse(t, mux, expectedResponse, assert.Contains)
 }
 
 func TestIncAnalysisRunReconcile(t *testing.T) {
@@ -184,7 +186,7 @@ analysis_run_reconcile_count{name="ar-test",namespace="ar-namespace"} 1`
 		},
 	}
 	metricsServ.IncAnalysisRunReconcile(ar, time.Millisecond)
-	testHttpResponse(t, metricsServ.Handler, expectedResponse)
+	testHttpResponse(t, metricsServ.Handler, expectedResponse, assert.Contains)
 }
 
 func TestAnalysisTemplateDescribe(t *testing.T) {
@@ -206,5 +208,5 @@ analysis_template_metric_info{metric="web-metric-2",name="http-benchmark-test",n
 	registry.MustRegister(NewAnalysisRunCollector(serverCfg.AnalysisRunLister, serverCfg.AnalysisTemplateLister, serverCfg.ClusterAnalysisTemplateLister))
 	mux := http.NewServeMux()
 	mux.Handle(MetricsPath, promhttp.HandlerFor(registry, promhttp.HandlerOpts{}))
-	testHttpResponse(t, mux, expectedResponse)
+	testHttpResponse(t, mux, expectedResponse, assert.Contains)
 }
diff --git a/controller/metrics/client_test.go b/controller/metrics/client_test.go
@@ -3,6 +3,8 @@ package metrics
 import (
 	"testing"
 
+	"github.com/stretchr/testify/assert"
+
 	"github.com/argoproj/pkg/kubeclientmetrics"
 )
 
@@ -24,5 +26,5 @@ func TestIncKubernetesRequest(t *testing.T) {
 		Verb:       kubeclientmetrics.Unknown,
 		StatusCode: 200,
 	})
-	testHttpResponse(t, metricsServ.Handler, expectedKubernetesRequest)
+	testHttpResponse(t, metricsServ.Handler, expectedKubernetesRequest, assert.Contains)
 }
diff --git a/controller/metrics/experiment_test.go b/controller/metrics/experiment_test.go
@@ -5,6 +5,8 @@ import (
 	"testing"
 	"time"
 
+	"github.com/stretchr/testify/assert"
+
 	"github.com/ghodss/yaml"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
@@ -133,7 +135,7 @@ func testExperimentDescribe(t *testing.T, fakeExperiment string, expectedRespons
 	registry.MustRegister(NewExperimentCollector(config.ExperimentLister))
 	mux := http.NewServeMux()
 	mux.Handle(MetricsPath, promhttp.HandlerFor(registry, promhttp.HandlerOpts{}))
-	testHttpResponse(t, mux, expectedResponse)
+	testHttpResponse(t, mux, expectedResponse, assert.Contains)
 }
 
 func TestIncExperimentReconcile(t *testing.T) {
@@ -156,5 +158,5 @@ experiment_reconcile_count{name="ex-test",namespace="ex-namespace"} 1`
 		},
 	}
 	metricsServ.IncExperimentReconcile(ex, time.Millisecond)
-	testHttpResponse(t, metricsServ.Handler, expectedResponse)
+	testHttpResponse(t, metricsServ.Handler, expectedResponse, assert.Contains)
 }