inline analysis doesnt wait for all metrics to run

[harikrongali]

Summary

Analysis run doesnt run metric that isnt started but other metrics completed by that time.
following example provides the scenario where one of the metric completed the analysis and posts successful but the other metric isn't started as it has delay. Analysis run completes without running delayed metric.

kind: AnalysisTemplate
metadata:
  name: success-rate-sample
spec:
  args:
  - name: namespace
  - name: stable-hash
  - name: canary-hash
  - name: app
  - name: env
  - name: prometheus-port
  - name: success-rate-min-percentage
  - name: initial-delay
    value: "5m"
  - name: analysis-interval
    value: "300s"
  - name: analysis-runs
    value: "36"
  - name: failure-limit
    value: "4"
  - name: inconclusive-limit
    value: "4"
  - name: secret-url
    valueFrom:
      secretKeyRef:
        name: example-secret
        key: secretUrl  
  metrics:
    - name: webmetric
      successCondition: result == 'It worked!'
      initialDelay: 60s
      count: 3
      interval: 5s
      failureLimit: 1
      provider:
        web:
          # placeholders are resolved when an AnalysisRun is created
          url: "{{args.secret-url}}"
          jsonPath: "{$.message}"
    - name: webmetric2
      successCondition: result == 'It worked!'
      count: 3
      interval: 5s
      failureLimit: 1
      provider:
        web:
          # placeholders are resolved when an AnalysisRun is created
          url: "{{args.secret-url}}"
          jsonPath: "{$.message}"

Diagnostics

What version of Argo Rollouts are you running?
1.0.2

# Paste the logs from the rollout controller

# Logs for the entire controller:
kubectl logs -n argo-rollouts deployment/argo-rollouts

# Logs for a specific rollout:
kubectl logs -n argo-rollouts deployment/argo-rollouts | grep rollout=<ROLLOUTNAME>

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[harikrongali]

working on the ticket