Istio traffic weight error

[giulianob]

Describe the bug

Error in 1.6.1 that is not happening in 1.6.0. This happens in the setHeaderRoute step below. See slack chat for some more details: https://cloud-native.slack.com/archives/C01U781DW2E/p1698875164861259

time="2023-11-01T21:28:37Z" level=info msg="Event(v1.ObjectReference{Kind:"Rollout", Namespace:"gbarberi-dp", Name:"frontend", UID:"abbea596-f2ed-44a8-95cb-37191c4caea1", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"173316067", FieldPath:""}): type: 'Warning' reason: 'TrafficRoutingError' admission webhook "validation.istio.io" denied the request: configuration is invalid: total destination weight = 0"

To Reproduce

Templates being used:

apiVersion: v1
kind: Service
metadata:
  name: {{ .Chart.Name }}
  annotations:
    networking.istio.io/exportTo: '.,ingress-system'
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
    targetPort: 80
  selector:
    app: {{ .Chart.Name }}
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: {{ .Chart.Name }}-vsvc
spec:
  exportTo:
    - .
    - ingress-system
  hosts:
    - frontend.{{ .Release.Namespace }}.svc.cluster.local
  http:
    - name: http-primary
      route:
        - destination:
            host: {{ .Chart.Name }}
            subset: stable
          weight: 100
        - destination:
            host: {{ .Chart.Name }}
            subset: canary
          weight: 0
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: {{ .Chart.Name }}-destrule
spec:
  host: {{ .Chart.Name }}
  exportTo:
    - .
    - ingress-system
  subsets:
    - name: canary
      labels:        # labels will be injected with canary rollouts-pod-template-hash value
        app: {{ .Chart.Name }}
    - name: stable
      labels:        # labels will be injected with stable rollouts-pod-template-hash value
        app: {{ .Chart.Name }}
---
kind: AnalysisTemplate
apiVersion: argoproj.io/v1alpha1
metadata:
  name: load-test
spec:
  args:
    - name: host
  metrics:
    - name: wrk
      initialDelay: 5s
      failureLimit: 0
      count: 1
      provider:
        job:
          spec:
            backoffLimit: 0
            ttlSecondsAfterFinished: 3600
            template:
              metadata:
                annotations:
                  sidecar.istio.io/inject: "false"
              spec:
                restartPolicy: Never
                containers:
                  - name: load-tester
                    image: argoproj/load-tester:latest
                    command: [ sh, -c, -x, -e ]
                    args:
                      - {{`|
                        wrk -t5 -c5 -d15s -s report.lua "https://{{ args.host }}/test" -H "subset: canary"
                        jq -e '.errors_ratio < 0.1' report.json`}}
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: {{ .Chart.Name }}
spec:
  replicas: {{ .Values.replicas | default 1 }}
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: {{ .Chart.Name }}
  strategy:
    canary:
      abortScaleDownDelaySeconds: 10
      scaleDownDelaySeconds: 10
      stableMetadata:
        labels:
          role: stable
      canaryMetadata:
        labels:
          role: canary
      trafficRouting:
        managedRoutes:
          - name: canary-header-route
        istio:
          virtualServices:
            - name: {{ .Chart.Name }}-vsvc
              routes:
                - http-primary
          destinationRule:
            name: {{ .Chart.Name }}-destrule
            canarySubsetName: canary
            stableSubsetName: stable
      steps:
        # Scale canary to 1 replica, cx traffic will not be routed to the canary yet
        - setCanaryScale: { replicas: 1 }
        # Forward only traffic with "subset: canary" header to canary
        - setHeaderRoute:
            name: canary-header-route
            match:
              - headerName: subset
                headerValue:
                  prefix: canary
        # For demo, pause until manual resume
        - pause: { }
        # Run a test against canary
        - analysis:
            templates:
              - templateName: load-test
            args:
              - name: host
                value: {{ template "frontend.ingress-host" . }}
        # Disable header routing to canary
        - setHeaderRoute:
            name: canary-header-route
        # Scale canary to matching the canary traffic weight (resets to default behavior)
        - setCanaryScale: { matchTrafficWeight: true }
        # Set weight to 25%
        - setWeight: 25
        # Wait 30s
        - pause: {duration: 30s}
        # Set weight to 75%
        - setWeight: 75
        # Pause for 30s
        - pause: {duration: 30s}
  template:
    metadata:
      labels:
        app: {{ .Chart.Name }}
    spec:
      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.imageTag -}}"
        imagePullPolicy: Always
        ports:
        - containerPort: 80
        startupProbe:
          httpGet:
            path: /health/startup
            port: 80
          initialDelaySeconds: 10 # Delay after service starts and startup probe begins
          periodSeconds: 3 # Time between checks
          failureThreshold: 30 # Restart container if probe fails this many times
          timeoutSeconds: 5 # Timeout of each check
        livenessProbe:
          httpGet:
            path: /healthz
            port: 80
          initialDelaySeconds: 5 # Delay after service starts and liveness probe begins
          periodSeconds: 10 # Time between checks
          failureThreshold: 3 # Restart container if probe fails this many times
          timeoutSeconds: 3 # Timeout of each check

Expected behavior

Version

1.6.1

Logs

# Paste the logs from the rollout controller

**I rolled back to 1.6.0. I need to upgrade again to 1.6.1 to get full logs. However, I did not see anything else in the logs that was useful it just kept retrying with the error above when trying to reconcile.**

# Logs for the entire controller:
kubectl logs -n argo-rollouts deployment/argo-rollouts

# Logs for a specific rollout:
kubectl logs -n argo-rollouts deployment/argo-rollouts | grep rollout=<ROLLOUTNAME

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.