Unable to substitute parameters for metric

[motybz]

First of all, thank you for a great framework!

Summary

We are using Argo 3.2.9
We are exporting metrics to Prometheus and it seems that part of the variables is not resolved.
The problematic metrics are {{exitCode}} and {{resourcesDuration.cpu}}
The errors are:
unable to substitute parameters for metric 'step_result_counter': failed to resolve {{exitCode}}
unable to substitute parameters for metric 'cpu_duration_gauge': failed to resolve {{resourcesDuration.cpu}}

Diagnostics

Not working metric

prometheus:
    - name: step_result_counter
      help: "Count of step execution by result status"
      labels:
        - key: name
          value: pre-processor
        - key: run
          value: "{{workflow.name}}"
        - key: status
          value: "{{status}}"
        **- key: exit_code
          value: "{{exitCode}}"**
      counter:
        value: "1"

Not working metrics

prometheus:
      - name: cpu_duration_gauge
        labels:
          - key: run
            value: "{{workflow.name}}"
          - key: name
            value: pre-processor
        help: "Step CPU duration gauge by name"
        gauge:
          value: "{{resourcesDuration.cpu}}"  

Working metric

prometheus:
    - name: step_result_counter
      help: "Count of step execution by result status"
      labels:
        - key: name
          value: pre-processor
        - key: run
          value: "{{workflow.name}}"
        - key: status
          value: "{{status}}"
      counter:
        value: "1"

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[sarabala1979]

@dpadhiar can you take a look?

[dpadhiar]

@motybz Hi, could you provide the workflow YAML so we may reproduce the bug? Thank you.

[nikita-akuity]

The issue appears in 3.4.4.
Workflow to reproduce:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: test-runner
  namespace: test
spec:
  entrypoint: main
  podGC:
    strategy: OnWorkflowSuccess
  serviceAccountName: operate-workflow-sa
  templates:
  - name: main
    steps:
    - - continueOn:
          error: true
          failed: true
        name: runTest
        template: run-test
  - container:
      name: runner
      image: alpine
      args:
      - exit 1
      command:
      - sh
      - -c
    metrics:
      prometheus:
      - counter:
          value: "1"
        help: Count of runs by exit code
        labels:
        - key: exit_code
          value: '{{exitCode}}'
        name: runs_exit_status_counter
    name: run-test
    retryStrategy:
      limit: "1"

workflow-controller log:

time="2023-01-05T00:28:46.457Z" level=info msg="node changed" namespace=test new.message="Error (exit code 1)" new.phase=Failed new.progress=0/1 nodeID=test-runner-cc48l-3076541373 old.message= old.phase=Pending old.progress=0/1 workflow=test-runner-cc48l
time="2023-01-05T00:28:56.482Z" level=info msg="node changed" namespace=test new.message="Error (exit code 1)" new.phase=Failed new.progress=0/1 nodeID=test-runner-cc48l-3412240848 old.message= old.phase=Pending old.progress=0/1 workflow=test-runner-cc48l
time="2023-01-05T00:28:56.482Z" level=error msg="unable to substitute parameters for metric 'runs_exit_status_counter': failed to resolve {{exitCode}}" namespace=test workflow=test-runner-cc48l

When exit code is 0 or retryStrategy is omitted, then {{exitCode}} resolves successfully and the error does not appear.

[jiachengxu]

I have repro the issue and will work on a fix for this.