optional property does not work with azure artifacts

[viploveb]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

Expected behaviour -

When optional is set to true then the workflow shouldn't error out even if the azure artifact isn't present.

Actual behaviour -

In case of azure artifact this doesn’t work as intended. I tried setting optional to true for an input artifact when the file is not present, it threw an error.
Error message -
Error (exit code 1): artifact file failed to load: unable to download blob test-artifact/s3-artifact/no-file.txt: ===== RESPONSE ERROR (ErrorCode=BlobNotFound) ===== Description=, Details: (none)

This works in case of s3 artifacts, when I set it to true and the file is not present it does not throw an error. (expected behaviour)

Version

v3.4.7

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: s3-azure-artifact
spec:
  entrypoint: main
  templates:
  - dag:
      tasks:
      - name: generate
        template: generate
      - depends: generate
        name: consume
        template: consume
    name: main
  - container:
      args:
      - echo hello > /mnt/file.txt; ls /mnt; cat /mnt/file.txt
      command:
      - bash
      - -c
      image: argoproj/argosay:v2
    name: generate
    outputs:
      artifacts:
      - archive:
          none: {}
        azure:
          blob: test-artifact/s3-artifact/file.txt
        name: file
        optional: true
        path: /mnt/file.txt
  - container:
      args:
      - cat
      - /tmp/file.txt
      command:
      - bash
      - -c
      image: argoproj/argosay:v2
    inputs:
      artifacts:
      - archive:
          none: {}
        azure:
          blob: test-artifact/s3-artifact/no-file.txt
        name: file
        optional: true
        path: /tmp/file.txt
    name: consume

Logs from the workflow controller

time="2023-06-05T06:33:43Z" level=info msg=cpu numCPU=8
time="2023-06-05T06:33:43Z" level=info msg=process pid=1
time="2023-06-05T06:33:43Z" level=info msg=version version=v0.7.0
time="2023-06-05T06:33:43Z" level=info msg="scaling config" defaultPeekDelay=4m0s defaultScalingDelay=1m0s
time="2023-06-05T06:33:43Z" level=info msg="reconciler config" imageFormat="ghcr.io/atlanhq/%s:v0.7.0" imagePullSecrets="[\"github-docker-registry\"]" pullPolicy="\"\"" runnerImage="ghcr.io/atlanhq/dataflow-runner:v0.7.0" updateInterval=15s
time="2023-06-05T06:33:44Z" level=info msg="metrics server is starting to listen" addr="127.0.0.1:9090" logger=controller-runtime.metrics
time="2023-06-05T06:33:44Z" level=info msg="starting manager" logger=setup
I0605 06:33:44.265445       1 leaderelection.go:243] attempting to acquire leader lease argo/1c03be80.my.domain...
time="2023-06-05T06:33:44Z" level=info msg="starting metrics server" logger=controller-runtime.manager path=/metrics
time="2023-06-05T06:33:44Z" level=info msg="started metrics cache worker 4"
time="2023-06-05T06:33:44Z" level=info msg="started metrics cache worker 1"
time="2023-06-05T06:33:44Z" level=info msg="started metrics cache worker 5"
time="2023-06-05T06:33:44Z" level=info msg="started metrics cache worker 2"
time="2023-06-05T06:33:44Z" level=info msg="started metrics cache worker 3"
I0605 06:34:01.691415       1 leaderelection.go:253] successfully acquired lease argo/1c03be80.my.domain
time="2023-06-05T06:34:01Z" level=info msg="Starting EventSource" logger=controller-runtime.manager.controller.pipeline reconciler group=dataflow.argoproj.io reconciler kind=Pipeline source="{\"Type\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{},\"status\":{\"lastUpdated\":null}}}"
time="2023-06-05T06:34:01Z" level=info msg="Starting EventSource" logger=controller-runtime.manager.controller.step reconciler group=dataflow.argoproj.io reconciler kind=Step source="{\"Type\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"name\":\"\",\"scale\":{},\"sidecar\":{\"resources\":{}}},\"status\":{\"phase\":\"\",\"replicas\":0,\"lastScaledAt\":null}}}"
time="2023-06-05T06:34:01Z" level=info msg="Starting EventSource" logger=controller-runtime.manager.controller.pipeline reconciler group=dataflow.argoproj.io reconciler kind=Pipeline source="{\"Type\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"name\":\"\",\"scale\":{},\"sidecar\":{\"resources\":{}}},\"status\":{\"phase\":\"\",\"replicas\":0,\"lastScaledAt\":null}}}"
time="2023-06-05T06:34:01Z" level=info msg="Starting Controller" logger=controller-runtime.manager.controller.pipeline reconciler group=dataflow.argoproj.io reconciler kind=Pipeline
time="2023-06-05T06:34:01Z" level=info msg="Starting workers" logger=controller-runtime.manager.controller.pipeline reconciler group=dataflow.argoproj.io reconciler kind=Pipeline worker count=1
time="2023-06-05T06:34:01Z" level=info msg="Starting EventSource" logger=controller-runtime.manager.controller.step reconciler group=dataflow.argoproj.io reconciler kind=Step source="{\"Type\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null},\"status\":{}}}"
time="2023-06-05T06:34:01Z" level=info msg="Starting EventSource" logger=controller-runtime.manager.controller.step reconciler group=dataflow.argoproj.io reconciler kind=Step source="{\"Type\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{},\"status\":{\"loadBalancer\":{}}}}"
time="2023-06-05T06:34:01Z" level=info msg="Starting Controller" logger=controller-runtime.manager.controller.step reconciler group=dataflow.argoproj.io reconciler kind=Step
time="2023-06-05T06:34:01Z" level=info msg="Starting workers" logger=controller-runtime.manager.controller.step reconciler group=dataflow.argoproj.io reconciler kind=Step worker count=1

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[viploveb]

Hey @liangyuanpeng, thanks for fixing it. Just wanted to confirm when can we expect the fix to release?

[terrytangyuan]

Had to retrigger the CI. Once it’s merged, you can try it in the latest image tag.