Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollback a canary deployment when using workoadRef #1254

Closed
sltet opened this issue Jun 7, 2021 · 7 comments · Fixed by #1275
Closed

Rollback a canary deployment when using workoadRef #1254

sltet opened this issue Jun 7, 2021 · 7 comments · Fixed by #1275
Labels
bug Something isn't working

Comments

@sltet
Copy link

sltet commented Jun 7, 2021

Summary

I'm trying to do a rollback using argo-rollouts and the workloadRef functionality but it seems broken.

here is my rollout:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rq
spec:
  replicas: 5
  workloadRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rq
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: {}
        - setWeight: 40
        - pause: {duration: 40s}
        - setWeight: 60
        - pause: {duration: 20s}
        - setWeight: 80
        - pause: {duration: 20s}

here is the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rq
  labels:
    helm.sh/chart: rq-0.1.0
    app.kubernetes.io/name: rq
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/version: "0.1.0"
    app.kubernetes.io/managed-by: Helm
spec:
  revisionHistoryLimit: 2
  replicas: 0
  selector:
    matchLabels:
      app.kubernetes.io/name: rq
      app.kubernetes.io/instance: RELEASE-NAME
  template:
    metadata:
      labels:
        app.kubernetes.io/name: rq
        app.kubernetes.io/instance: RELEASE-NAME
    spec:
      serviceAccountName: default
      securityContext:
        {}
      containers:

When i try to rollback using the cli, nothing still happens in the cluster even when i get an Ok message:

kubectl argo rollouts undo rq --to-revision=2
skipped rollback (current template already matches revision 2)

kubectl argo rollouts undo rq --to-revision=1
rollout 'rq' undo

When i try rollback using the dashboard this is the error i get:

{"code":2,"message":"rollout.argoproj.io \"rq\" not found"}

Diagnostics

kubectl-argo-rollouts: v1.0.1+ff65b39
  BuildDate: 2021-05-26T01:02:54Z
  GitCommit: ff65b39f21a0be79744d45d7169de6ee84e991a6
  GitTreeState: clean
  GoVersion: go1.16.3
  Compiler: gc
  Platform: linux/amd64

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

@sltet sltet added the bug Something isn't working label Jun 7, 2021
@huikang
Copy link
Member

huikang commented Jun 7, 2021

Hi, @sltet , should revisionHistoryLimit: 2 be in the rollout spec, instead of deployment? Also could you paste the full history of command you use to update and rollback? I can try reproduce this on my end. Thanks.

@sltet
Copy link
Author

sltet commented Jun 8, 2021

Here is the current state:

rollback-rollout-revlimit

rollout undo first attempt:

kubectl argo rollouts undo rq --to-revision=2
rollout 'rq' undo

rollout undo second attempt:

kubectl argo rollouts undo rq --to-revision=2
skipped rollback (current template already matches revision 2)

Notes:

  • Updates are done using the helm upgrade command
  • Nothing changed in the cluster after the first and the second attempt
  • Not working through the dashboard
  • revisionHistoryLimit: 2 was added in the rollout spec

@huikang
Copy link
Member

huikang commented Jun 9, 2021

@sltet , thanks for your help. Are you saying the cli rollback is working, while the dashboard doesn't?

@sltet
Copy link
Author

sltet commented Jun 9, 2021

@huikang none of them are working. The cli rollback pretend to work by giving an OK message ( rollout 'rq' undo ) but nothing happens in the cluster

@huikang
Copy link
Member

huikang commented Jun 9, 2021

Confirmed. I will look into this issue.

@huikang
Copy link
Member

huikang commented Jun 9, 2021

So I think the root cause is that during forward update, the rollout's spec.template is empty because the template is referencing the part of the deployment. However, on a rollback, rollout's spec.template somehow is filled by the controller, which results in the error of invalidSpec.

Will send a PR to fix it.

@jessesuen
Copy link
Member

jessesuen commented Jun 9, 2021

I think we need to rethink what the undo functionality for rollouts (Rollback button in the dashboard) means when used with workloadRef.

Currently, it doesn't make sense when using workloadRef, because what has to happen is for the referenced object needs to be reverted, and the Rollout spec wouldn't actually change.

I think the correct approach is: when user performs kubectl argo rollouts undo against a Rollout using workloadRef, the CLI will actually modify the Deployment object rather than the Rollout object.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants