Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

blue-green rollouts could pause prematurely during prePromotionAnalysis #990

Closed
jessesuen opened this issue Feb 13, 2021 · 2 comments · Fixed by #1007
Closed

blue-green rollouts could pause prematurely during prePromotionAnalysis #990

jessesuen opened this issue Feb 13, 2021 · 2 comments · Fixed by #1007
Labels
bug Something isn't working

Comments

@jessesuen
Copy link
Member

Summary

The TestBlueGreenAnalysis intermittently fails with the following:

=== RUN   TestAnalysisSuite/TestBlueGreenAnalysis
time="2021-02-12T20:53:50-08:00" level=info msg="Deleting e2e-test-name=TestBlueGreenAnalysis"
time="2021-02-12T20:53:51-08:00" level=info msg="service/bluegreen-analysis-active created\n" rollout=bluegreen-analysis
time="2021-02-12T20:53:51-08:00" level=info msg="service/bluegreen-analysis-preview created\n" rollout=bluegreen-analysis
time="2021-02-12T20:53:51-08:00" level=info msg="rollout.argoproj.io/bluegreen-analysis created\n" rollout=bluegreen-analysis
time="2021-02-12T20:53:51-08:00" level=info msg="Waiting for condition: status=Healthy" rollout=bluegreen-analysis
time="2021-02-12T20:53:53-08:00" level=info msg="Condition 'status=Healthy' met after 1s" rollout=bluegreen-analysis
time="2021-02-12T20:53:53-08:00" level=info msg="AnalysisRun expectation 'analysisrun count == 0' met" rollout=bluegreen-analysis
time="2021-02-12T20:53:53-08:00" level=info msg="Updated rollout spec: {\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"image\":\"nginx:1.19-alpine\",\"name\":\"bluegreen-analysis\",\"resources\":{\"requests\":{\"cpu\":\"4m\",\"memory\":\"16Mi\"}}}]}}}}" rollout=bluegreen-analysis
time="2021-02-12T20:53:53-08:00" level=info msg="Waiting for condition: status=Progressing" rollout=bluegreen-analysis
time="2021-02-12T20:53:53-08:00" level=info msg="Condition 'status=Progressing' met after 0s" rollout=bluegreen-analysis
time="2021-02-12T20:53:53-08:00" level=info msg="Waiting for condition: status=Paused" rollout=bluegreen-analysis
time="2021-02-12T20:53:55-08:00" level=info msg="Condition 'status=Paused' met after 2s" rollout=bluegreen-analysis
time="2021-02-12T20:53:55-08:00" level=info msg="Expectation: active revision == '1' met" rollout=bluegreen-analysis
time="2021-02-12T20:53:55-08:00" level=info msg="Expectation: preview revision == '2' met" rollout=bluegreen-analysis
time="2021-02-12T20:53:57-08:00" level=info msg="Promoted rollout" rollout=bluegreen-analysis
time="2021-02-12T20:53:57-08:00" level=info msg="Waiting for condition: active revision=2" rollout=bluegreen-analysis
    when.go:313: timeout after 1m0s waiting for condition active revision=2
Name:            bluegreen-analysis
Namespace:       test
Status:          ॥ Paused
Message:         BlueGreenPause
Strategy:        BlueGreen
Images:          nginx:1.19-alpine (active, preview, stable)
Replicas:
  Desired:       1
  Current:       2
  Updated:       1
  Ready:         1
  Available:     1

NAME                                                         KIND         STATUS        AGE  INFO
⟳ bluegreen-analysis                                         Rollout      ॥ Paused      64s
├──# revision:2
│  ├──⧉ bluegreen-analysis-754987cd65                        ReplicaSet   ✔ Healthy     63s  preview
│  │  └──□ bluegreen-analysis-754987cd65-xd77s               Pod          ✔ Running     63s  ready:1/1
│  └──α bluegreen-analysis-754987cd65-2-pre                  AnalysisRun  ✔ Successful  61s  ✔ 1
│     └──⊞ 70beec65-6065-424b-88ce-24fcfa00b895.sleep-job.1  Job          ✔ Successful  62s
└──# revision:1
   └──⧉ bluegreen-analysis-6cc94d85f                         ReplicaSet   ✔ Healthy     64s  stable,active
      └──□ bluegreen-analysis-6cc94d85f-q56r9                Pod          ✔ Running     64s  ready:1/1

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    rollout.argoproj.io/revision: "2"
  creationTimestamp: "2021-02-13T04:53:53Z"
  generation: 3
  labels:
    argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e
    e2e-test-name: TestBlueGreenAnalysis
  name: bluegreen-analysis
  namespace: test
  resourceVersion: "506592"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/test/rollouts/bluegreen-analysis
  uid: 0124f2e8-b8df-43e5-ae4e-ec2e2684be51
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bluegreen-analysis
  strategy:
    blueGreen:
      activeService: bluegreen-analysis-active
      autoPromotionEnabled: false
      postPromotionAnalysis:
        args:
        - name: duration
          value: "5"
        templates:
        - templateName: sleep-job
      prePromotionAnalysis:
        templates:
        - templateName: sleep-job
      previewService: bluegreen-analysis-preview
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: bluegreen-analysis
    spec:
      containers:
      - image: nginx:1.19-alpine
        name: bluegreen-analysis
        resources:
          requests:
            cpu: 4m
            memory: 16Mi
status:
  HPAReplicas: 1
  availableReplicas: 1
  blueGreen:
    activeSelector: 6cc94d85f
    prePromotionAnalysisRun: bluegreen-analysis-754987cd65-2-pre
    prePromotionAnalysisRunStatus:
      name: bluegreen-analysis-754987cd65-2-pre
      status: Successful
    previewSelector: 754987cd65
  canary: {}
  conditions:
  - lastTransitionTime: "2021-02-13T04:53:53Z"
    lastUpdateTime: "2021-02-13T04:53:53Z"
    message: Rollout has minimum availability
    reason: AvailableReason
    status: "True"
    type: Available
  - lastTransitionTime: "2021-02-13T04:53:55Z"
    lastUpdateTime: "2021-02-13T04:53:57Z"
    message: Rollout is paused
    reason: RolloutPaused
    status: Unknown
    type: Progressing
  controllerPause: true
  currentPodHash: 754987cd65
  observedGeneration: "3"
  pauseConditions:
  - reason: BlueGreenPause
    startTime: "2021-02-13T04:53:57Z"
  readyReplicas: 1
  replicas: 2
  selector: app=bluegreen-analysis,rollouts-pod-template-hash=6cc94d85f
  stableRS: 6cc94d85f
  updatedReplicas: 1

I managed to capture controller logs around the time of test:

INFO[2021-02-12T20:53:53-08:00] Switched selector for service 'bluegreen-analysis-active' from '' to '6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"test", Name:"bluegreen-analysis", UID:"0124f2e8-b8df-43e5-ae4e-ec2e2684be51", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"506520", FieldPath:""}): type: 'Normal' reason: 'SwitchService' Switched selector for service 'bluegreen-analysis-active' from '' to '6cc94d85f'
INFO[2021-02-12T20:53:53-08:00] Updating active selector ( -> 6cc94d85f)      namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Patched: {"status":{"availableReplicas":1,"blueGreen":{"activeSelector":"6cc94d85f"},"conditions":[{"lastTransitionTime":"2021-02-13T04:53:51Z","lastUpdateTime":"2021-02-13T04:53:53Z","message":"ReplicaSet \"bluegreen-analysis-6cc94d85f\" has successfully progressed.","reason":"NewReplicaSetAvailable","status":"True","type":"Progressing"},{"lastTransitionTime":"2021-02-13T04:53:53Z","lastUpdateTime":"2021-02-13T04:53:53Z","message":"Rollout has minimum availability","reason":"AvailableReason","status":"True","type":"Available"}],"readyReplicas":1,"selector":"app=bluegreen-analysis,rollouts-pod-template-hash=6cc94d85f"}}  generation=2 namespace=test resourceVersion=506520 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=37.300647999999995
INFO[2021-02-12T20:53:53-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] No status changes. Skipping patch             generation=2 namespace=test resourceVersion=506533 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=2.1718129999999998
INFO[2021-02-12T20:53:53-08:00] Service index keys modified. before: [test/bluegreen-analysis-active test/bluegreen-analysis-preview], after: [test/bluegreen-analysis-preview test/bluegreen-analysis-active]  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] syncing service                               namespace=test rollout=bluegreen-analysis service=bluegreen-analysis-preview
INFO[2021-02-12T20:53:53-08:00] syncing service                               namespace=test rollout=bluegreen-analysis service=bluegreen-analysis-active
INFO[2021-02-12T20:53:53-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Updating replica set 'bluegreen-analysis-754987cd65' revision from 0 to 2  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Created ReplicaSet bluegreen-analysis-754987cd65  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Enqueueing parent of test/bluegreen-analysis-754987cd65: Rollout test/bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"test", Name:"bluegreen-analysis", UID:"0124f2e8-b8df-43e5-ae4e-ec2e2684be51", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"506534", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set bluegreen-analysis-754987cd65 to 1
INFO[2021-02-12T20:53:53-08:00] Enqueueing parent of test/bluegreen-analysis-754987cd65: Rollout test/bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Updated rollout revision to 2                 namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Set rollout condition: &{Progressing True 2021-02-12 20:53:53.291487 -0800 PST m=+37.589894298 2021-02-12 20:53:53.291487 -0800 PST m=+37.589894410 NewReplicaSetCreated Created new replica set "bluegreen-analysis-754987cd65"}  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] New pod template or template change detected  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Pod template change detected (new: 754987cd65, old: 6cc94d85f)  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Enqueueing parent of test/bluegreen-analysis-754987cd65: Rollout test/bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Patched: {"status":{"conditions":[{"lastTransitionTime":"2021-02-13T04:53:53Z","lastUpdateTime":"2021-02-13T04:53:53Z","message":"Rollout has minimum availability","reason":"AvailableReason","status":"True","type":"Available"},{"lastTransitionTime":"2021-02-13T04:53:51Z","lastUpdateTime":"2021-02-13T04:53:53Z","message":"ReplicaSet \"bluegreen-analysis-754987cd65\" is progressing.","reason":"ReplicaSetUpdated","status":"True","type":"Progressing"}],"currentPodHash":"754987cd65","observedGeneration":"3","updatedReplicas":null}}  generation=3 namespace=test resourceVersion=506543 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=175.398146
INFO[2021-02-12T20:53:53-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] syncing service                               namespace=test rollout=bluegreen-analysis service=bluegreen-analysis-preview
INFO[2021-02-12T20:53:53-08:00] Switched selector for service 'bluegreen-analysis-preview' from '6cc94d85f' to '754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] New RS 'bluegreen-analysis-754987cd65' is not ready to pause  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] New RS 'bluegreen-analysis-754987cd65' is not fully saturated  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"test", Name:"bluegreen-analysis", UID:"0124f2e8-b8df-43e5-ae4e-ec2e2684be51", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"506545", FieldPath:""}): type: 'Normal' reason: 'SwitchService' Switched selector for service 'bluegreen-analysis-preview' from '6cc94d85f' to '754987cd65'
INFO[2021-02-12T20:53:53-08:00] Updating preview selector (6cc94d85f -> 754987cd65)  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Patched: {"status":{"blueGreen":{"previewSelector":"754987cd65"},"replicas":2,"updatedReplicas":1}}  generation=3 namespace=test resourceVersion=506545 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Service index keys modified. before: [test/bluegreen-analysis-preview test/bluegreen-analysis-active], after: [test/bluegreen-analysis-active test/bluegreen-analysis-preview]  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=48.458026000000004
INFO[2021-02-12T20:53:53-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] New RS 'bluegreen-analysis-754987cd65' is not ready to pause  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] New RS 'bluegreen-analysis-754987cd65' is not fully saturated  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Timed out (false) [last progress check: 2021-02-12 20:53:53 -0800 PST - now: 2021-02-12 20:53:53.430025 -0800 PST m=+37.728431795]  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] No status changes. Skipping patch             generation=3 namespace=test resourceVersion=506549 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Queueing up rollout for a progress after 599s  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:53-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=1.7226240000000002
INFO[2021-02-12T20:53:55-08:00] Enqueueing parent of test/bluegreen-analysis-754987cd65: Rollout test/bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Enqueueing parent of test/bluegreen-analysis-754987cd65-2-pre: Rollout test/bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Started syncing Analysis at (2021-02-12 20:53:55.234557 -0800 PST m=+39.532962376)  analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:55-08:00] running initial measurement                   analysisrun=bluegreen-analysis-754987cd65-2-pre metric=sleep-job namespace=test
INFO[2021-02-12T20:53:55-08:00] taking 1 measurements                         analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:55-08:00] Created Pre Promotion AnalysisRun             analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"test", Name:"bluegreen-analysis", UID:"0124f2e8-b8df-43e5-ae4e-ec2e2684be51", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"506549", FieldPath:""}): type: 'Normal' reason: 'AnalysisRunStatusChange' PrePromotion Analysis Run 'bluegreen-analysis-754987cd65-2-pre' Status New: '' Previous: 'NoPreviousStatus'
INFO[2021-02-12T20:53:55-08:00] Timed out (false) [last progress check: 2021-02-12 20:53:53 -0800 PST - now: 2021-02-12 20:53:55.236039 -0800 PST m=+39.534444608]  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Adding pause reason BlueGreenPause with start time 2021-02-13T04:53:55Z  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] job test/70beec65-6065-424b-88ce-24fcfa00b895.sleep-job.1 created  analysisrun=bluegreen-analysis-754987cd65-2-pre metric=sleep-job namespace=test
INFO[2021-02-12T20:53:55-08:00] analysis transitioned from  -> Running        analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:55-08:00] Patched: {"status":{"blueGreen":{"prePromotionAnalysisRun":"bluegreen-analysis-754987cd65-2-pre","prePromotionAnalysisRunStatus":{"name":"bluegreen-analysis-754987cd65-2-pre","status":""}},"controllerPause":true,"pauseConditions":[{"reason":"BlueGreenPause","startTime":"2021-02-13T04:53:55Z"}]}}  generation=3 namespace=test resourceVersion=506549 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=26.598726
INFO[2021-02-12T20:53:55-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Enqueueing parent of test/bluegreen-analysis-754987cd65-2-pre: Rollout test/bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Patch status successfully                     analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:55-08:00] Reconciliation completed                      analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test time_ms=23.154522
INFO[2021-02-12T20:53:55-08:00] Started syncing Analysis at (2021-02-12 20:53:55.257753 -0800 PST m=+39.556158026)  analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:55-08:00] resuming in-progress measurement              analysisrun=bluegreen-analysis-754987cd65-2-pre metric=sleep-job namespace=test
INFO[2021-02-12T20:53:55-08:00] taking 1 measurements                         analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:55-08:00] No status changes. Skipping patch             analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:55-08:00] Reconciliation completed                      analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test time_ms=0.423896
INFO[2021-02-12T20:53:55-08:00] Patched conditions: {"status":{"conditions":[{"lastTransitionTime":"2021-02-13T04:53:53Z","lastUpdateTime":"2021-02-13T04:53:53Z","message":"Rollout has minimum availability","reason":"AvailableReason","status":"True","type":"Available"},{"lastTransitionTime":"2021-02-13T04:53:55Z","lastUpdateTime":"2021-02-13T04:53:55Z","message":"Rollout is paused","reason":"RolloutPaused","status":"Unknown","type":"Progressing"}]}}  generation=3 namespace=test resourceVersion=506563 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] No status changes. Skipping patch             generation=3 namespace=test resourceVersion=506563 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=81.19287
INFO[2021-02-12T20:53:55-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"test", Name:"bluegreen-analysis", UID:"0124f2e8-b8df-43e5-ae4e-ec2e2684be51", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"506571", FieldPath:""}): type: 'Normal' reason: 'AnalysisRunStatusChange' PrePromotion Analysis Run 'bluegreen-analysis-754987cd65-2-pre' Status New: 'Running' Previous: ''
INFO[2021-02-12T20:53:55-08:00] Patched: {"status":{"blueGreen":{"prePromotionAnalysisRunStatus":{"status":"Running"}}}}  generation=3 namespace=test resourceVersion=506571 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=39.127668
INFO[2021-02-12T20:53:55-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] No status changes. Skipping patch             generation=3 namespace=test resourceVersion=506574 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=5.069353
INFO[2021-02-12T20:53:55-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] No status changes. Skipping patch             generation=3 namespace=test resourceVersion=506574 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=2.136035
INFO[2021-02-12T20:53:57-08:00] Service index keys modified. before: [test/bluegreen-analysis-active test/bluegreen-analysis-preview], after: [test/bluegreen-analysis-preview test/bluegreen-analysis-active]  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] syncing service                               namespace=test rollout=bluegreen-analysis service=bluegreen-analysis-preview
INFO[2021-02-12T20:53:57-08:00] syncing service                               namespace=test rollout=bluegreen-analysis service=bluegreen-analysis-active
INFO[2021-02-12T20:53:57-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Patched conditions: {"status":{"conditions":[{"lastTransitionTime":"2021-02-13T04:53:53Z","lastUpdateTime":"2021-02-13T04:53:53Z","message":"Rollout has minimum availability","reason":"AvailableReason","status":"True","type":"Available"},{"lastTransitionTime":"2021-02-13T04:53:55Z","lastUpdateTime":"2021-02-13T04:53:57Z","message":"Rollout is resumed","reason":"RolloutResumed","status":"Unknown","type":"Progressing"}]}}  generation=3 namespace=test resourceVersion=506580 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Timed out (false) [last progress check: 2021-02-12 20:53:55 -0800 PST - now: 2021-02-12 20:53:57.382235 -0800 PST m=+41.680638621]  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Patched: {"status":{"controllerPause":null}}  generation=3 namespace=test resourceVersion=506580 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=46.595536
INFO[2021-02-12T20:53:57-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Timed out (false) [last progress check: 2021-02-12 20:53:57 -0800 PST - now: 2021-02-12 20:53:57.403495 -0800 PST m=+41.701898459]  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Adding pause reason BlueGreenPause with start time 2021-02-13T04:53:57Z  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Patched: {"status":{"controllerPause":true,"pauseConditions":[{"reason":"BlueGreenPause","startTime":"2021-02-13T04:53:57Z"}]}}  generation=3 namespace=test resourceVersion=506582 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=19.926436
INFO[2021-02-12T20:53:57-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Patched conditions: {"status":{"conditions":[{"lastTransitionTime":"2021-02-13T04:53:53Z","lastUpdateTime":"2021-02-13T04:53:53Z","message":"Rollout has minimum availability","reason":"AvailableReason","status":"True","type":"Available"},{"lastTransitionTime":"2021-02-13T04:53:55Z","lastUpdateTime":"2021-02-13T04:53:57Z","message":"Rollout is paused","reason":"RolloutPaused","status":"Unknown","type":"Progressing"}]}}  generation=3 namespace=test resourceVersion=506583 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] No status changes. Skipping patch             generation=3 namespace=test resourceVersion=506583 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=15.878232999999998
INFO[2021-02-12T20:53:57-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] No status changes. Skipping patch             generation=3 namespace=test resourceVersion=506584 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=2.47775
INFO[2021-02-12T20:53:57-08:00] Enqueueing parent of test/70beec65-6065-424b-88ce-24fcfa00b895.sleep-job.1: AnalysisRun test/bluegreen-analysis-754987cd65-2-pre
INFO[2021-02-12T20:53:57-08:00] Started syncing Analysis at (2021-02-12 20:53:57.472664 -0800 PST m=+41.771066938)  analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:57-08:00] resuming in-progress measurement              analysisrun=bluegreen-analysis-754987cd65-2-pre metric=sleep-job namespace=test
INFO[2021-02-12T20:53:57-08:00] taking 1 measurements                         analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:57-08:00] job test/70beec65-6065-424b-88ce-24fcfa00b895.sleep-job.1 completed: Successful  analysisrun=bluegreen-analysis-754987cd65-2-pre metric=sleep-job namespace=test
INFO[2021-02-12T20:53:57-08:00] measurement completed Successful              analysisrun=bluegreen-analysis-754987cd65-2-pre metric=sleep-job namespace=test
INFO[2021-02-12T20:53:57-08:00] metric assessed Successful: count (1) reached  metric=sleep-job
INFO[2021-02-12T20:53:57-08:00] metric transitioned from Running -> Successful  analysisrun=bluegreen-analysis-754987cd65-2-pre metric=sleep-job namespace=test
INFO[2021-02-12T20:53:57-08:00] analysis transitioned from Running -> Successful  analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:57-08:00] Event(v1.ObjectReference{Kind:"AnalysisRun", Namespace:"test", Name:"bluegreen-analysis-754987cd65-2-pre", UID:"70beec65-6065-424b-88ce-24fcfa00b895", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"506564", FieldPath:""}): type: 'Normal' reason: 'Complete' metric 'sleep-job' completed Successful
INFO[2021-02-12T20:53:57-08:00] Event(v1.ObjectReference{Kind:"AnalysisRun", Namespace:"test", Name:"bluegreen-analysis-754987cd65-2-pre", UID:"70beec65-6065-424b-88ce-24fcfa00b895", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"506564", FieldPath:""}): type: 'Normal' reason: 'Complete' analysis completed Successful
INFO[2021-02-12T20:53:57-08:00] Patch status successfully                     analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:57-08:00] Reconciliation completed                      analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test time_ms=8.925040000000001
INFO[2021-02-12T20:53:57-08:00] Started syncing Analysis at (2021-02-12 20:53:57.481692 -0800 PST m=+41.780095779)  analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:57-08:00] No status changes. Skipping patch             analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:57-08:00] Reconciliation completed                      analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test time_ms=0.307096
INFO[2021-02-12T20:53:57-08:00] Enqueueing parent of test/bluegreen-analysis-754987cd65-2-pre: Rollout test/bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"test", Name:"bluegreen-analysis", UID:"0124f2e8-b8df-43e5-ae4e-ec2e2684be51", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"506584", FieldPath:""}): type: 'Normal' reason: 'AnalysisRunStatusChange' PrePromotion Analysis Run 'bluegreen-analysis-754987cd65-2-pre' Status New: 'Successful' Previous: 'Running'
INFO[2021-02-12T20:53:57-08:00] Patched: {"status":{"blueGreen":{"prePromotionAnalysisRunStatus":{"status":"Successful"}}}}  generation=3 namespace=test resourceVersion=506584 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=17.74243
INFO[2021-02-12T20:53:57-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] No status changes. Skipping patch             generation=3 namespace=test resourceVersion=506592 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=1.6274639999999998
INFO[2021-02-12T20:54:57-08:00] rollout deleted                               namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:54:57-08:00] syncing service                               namespace=test rollout=bluegreen-analysis service=bluegreen-analysis-active
INFO[2021-02-12T20:54:57-08:00] syncing service                               namespace=test rollout=bluegreen-analysis service=bluegreen-analysis-preview
INFO[2021-02-12T20:54:57-08:00] rollout deleted                               namespace=test rollout=bluegreen-analysis

Diagnostics

What version of Argo Rollouts are you running? v1.0 c8eb953

# Paste the logs from the rollout controller

# Logs for the entire controller:
kubectl logs -n argo-rollouts deployment/argo-rollouts

# Logs for a specific rollout:
kubectl logs -n argo-rollouts deployment/argo-rollouts | grep rollout=<ROLLOUTNAME>

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.
@jessesuen jessesuen added the bug Something isn't working label Feb 13, 2021
@jessesuen
Copy link
Member Author

jessesuen commented Feb 13, 2021 •
edited
Loading

It appears the controller paused the rollout, unpaused it (as a result of the test promoting the rollout), and then paused it again:

INFO[2021-02-12T20:53:55-08:00] Adding pause reason BlueGreenPause with start time 2021-02-13T04:53:55Z  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:55-08:00] job test/70beec65-6065-424b-88ce-24fcfa00b895.sleep-job.1 created  analysisrun=bluegreen-analysis-754987cd65-2-pre metric=sleep-job namespace=test
INFO[2021-02-12T20:53:55-08:00] analysis transitioned from  -> Running        analysisrun=bluegreen-analysis-754987cd65-2-pre namespace=test
INFO[2021-02-12T20:53:55-08:00] Patched: {"status":{"blueGreen":{"prePromotionAnalysisRun":"bluegreen-analysis-754987cd65-2-pre","prePromotionAnalysisRunStatus":{"name":"bluegreen-analysis-754987cd65-2-pre","status":""}},"controllerPause":true,"pauseConditions":[{"reason":"BlueGreenPause","startTime":"2021-02-13T04:53:55Z"}]}}  generation=3 namespace=test resourceVersion=506549 rollout=bluegreen-analysis

INFO[2021-02-12T20:53:57-08:00] Started syncing rollout                       namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Patched conditions: {"status":{"conditions":[{"lastTransitionTime":"2021-02-13T04:53:53Z","lastUpdateTime":"2021-02-13T04:53:53Z","message":"Rollout has minimum availability","reason":"AvailableReason","status":"True","type":"Available"},{"lastTransitionTime":"2021-02-13T04:53:55Z","lastUpdateTime":"2021-02-13T04:53:57Z","message":"Rollout is resumed","reason":"RolloutResumed","status":"Unknown","type":"Progressing"}]}}  generation=3 namespace=test resourceVersion=506580 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling stable ReplicaSet 'bluegreen-analysis-6cc94d85f'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling new ReplicaSet 'bluegreen-analysis-754987cd65'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling preview service 'bluegreen-analysis-preview'  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling pause                             namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Pre Promotion Analysis            namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciling Post Promotion Analysis           namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Timed out (false) [last progress check: 2021-02-12 20:53:55 -0800 PST - now: 2021-02-12 20:53:57.382235 -0800 PST m=+41.680638621]  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Patched: {"status":{"controllerPause":null}}  generation=3 namespace=test resourceVersion=506580 rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Reconciliation completed                      namespace=test rollout=bluegreen-analysis time_ms=46.595536

INFO[2021-02-12T20:53:57-08:00] Adding pause reason BlueGreenPause with start time 2021-02-13T04:53:57Z  namespace=test rollout=bluegreen-analysis
INFO[2021-02-12T20:53:57-08:00] Patched: {"status":{"controllerPause":true,"pauseConditions":[{"reason":"BlueGreenPause","startTime":"2021-02-13T04:53:57Z"}]}}  generation=3 namespace=test resourceVersion=506582 rollout=bluegreen-analysis

@jessesuen
Copy link
Member Author

This is easily reproducible with the following modification to the test:

$ git diff test/e2e/analysis_test.go
diff --git a/test/e2e/analysis_test.go b/test/e2e/analysis_test.go
index 450f367..8f13f59 100644
--- a/test/e2e/analysis_test.go
+++ b/test/e2e/analysis_test.go
@@ -99,6 +99,9 @@ spec:
       prePromotionAnalysis:
         templates:
         - templateName: sleep-job
+        args:
+        - name: duration
+          value: "5"
       postPromotionAnalysis:
         templates:
         - templateName: sleep-job
@@ -149,6 +152,7 @@ spec:
                WaitForRolloutStatus("Progressing").
                WaitForRolloutStatus("Paused").
                Then().
+               ExpectAnalysisRunCount(1).
                ExpectActiveRevision("1").
                ExpectPreviewRevision("2").
                When().

It seems there is a real bug in that we should not have paused the rollout until prePromotionAnalysis completed

@jessesuen jessesuen changed the title E2e TestBlueGreenAnalysis is flakey blue-green rollouts could pause prematurely during prePromotionAnalysis Feb 24, 2021
@jessesuen jessesuen mentioned this issue Feb 24, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: blue-green rollouts could pause prematurely during prePromotionAnalysis jessesuen/argo-rollouts
1 participant
@jessesuen