Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exit Code on Windows not correct #11271

Closed
3 tasks done
cbuchli opened this issue Jun 27, 2023 · 0 comments · Fixed by #11276
Closed
3 tasks done

Exit Code on Windows not correct #11271

cbuchli opened this issue Jun 27, 2023 · 0 comments · Fixed by #11276
Labels
area/executor area/windows Windows Container support P3 Low priority type/bug

Comments

@cbuchli
Copy link
Contributor

cbuchli commented Jun 27, 2023

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

From the docs, I have the following Example of using the ExitCode in a Retry Expression like this asInt(lastRetry.exitCode). This doesn't work under Windows because the exit code always is 64 with the message Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>",

I would expect to be able to use expressions like this in Windows, for which the ExitCode would have to be correctly propagated there as well.

This is how the workflow looks like:

Name:                retry-script-qztfz
Namespace:           workflows
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Running
Conditions:
 PodRunning          True
Created:             Tue Jun 27 09:49:45 +0200 (2 minutes ago)
Started:             Tue Jun 27 09:49:45 +0200 (2 minutes ago)
Duration:            2 minutes 55 seconds
Progress:            0/3
ResourcesDuration:   10s*(1 cpu),10s*(100Mi memory)

STEP                   TEMPLATE      PODNAME                                     DURATION  MESSAGE
 ● retry-script-qztfz  main
 └───● retry           retry-script                                                        retryStrategy.expression evaluated to false
     ├─✖ retry(0)      retry-script  retry-script-qztfz-retry-script-477302956   25s       Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>
     ├─✖ retry(1)      retry-script  retry-script-qztfz-retry-script-3094361257  25s       Error (exit code 64): exit with non-zero code. exit-code: 1, error:<nil>
     └─● retry(2)      retry-script  retry-script-qztfz-retry-script-1324344185  45s

And later when it's done:

STEP                   TEMPLATE      PODNAME                                     DURATION  MESSAGE
 ✖ retry-script-qztfz  main                                                                child 'retry-script-qztfz-2233819039' failed
 └───✖ retry           retry-script                                                        No more retries left
     ├─✖ retry(0)      retry-script  retry-script-qztfz-retry-script-477302956   25s       Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>
     ├─✖ retry(1)      retry-script  retry-script-qztfz-retry-script-3094361257  25s       Error (exit code 64): exit with non-zero code. exit-code: 1, error:<nil>
     ├─✖ retry(2)      retry-script  retry-script-qztfz-retry-script-2691551306  25s       Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>
     ├─✖ retry(3)      retry-script  retry-script-qztfz-retry-script-3161177543  23s       Error (exit code 64): exit with non-zero code. exit-code: 1, error:<nil>
     ├─✖ retry(4)      retry-script  retry-script-qztfz-retry-script-2489925688  24s       Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>
     └─✖ retry(5)      retry-script  retry-script-qztfz-retry-script-543574789   26s       Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>

And here is some log output of the main containers:

retry-script-qztfz-retry-script-2272194143: time="2023-06-27T07:51:15.070Z" level=info msg="sub-process exited" argo=true error="<nil>"
retry-script-qztfz-retry-script-2272194143: Error: exit with non-zero code. exit-code: 1, error:<nil>
retry-script-qztfz-retry-script-2272194143: exit with non-zero code. exit-code: 1, error:<nil>
retry-script-qztfz-retry-script-125203364: time="2023-06-27T07:52:20.303Z" level=info msg="sub-process exited" argo=true error="<nil>"
retry-script-qztfz-retry-script-125203364: Error: exit with non-zero code. exit-code: 3, error:<nil>
retry-script-qztfz-retry-script-125203364: exit with non-zero code. exit-code: 3, error:<nil>
retry-script-qztfz-retry-script-594829601: time="2023-06-27T07:53:23.160Z" level=info msg="sub-process exited" argo=true error="<nil>"
retry-script-qztfz-retry-script-594829601: exit with non-zero code. exit-code: 1, error:<nil>
retry-script-qztfz-retry-script-594829601: Error: exit with non-zero code. exit-code: 1, error:<nil>
retry-script-qztfz-retry-script-593990958: time="2023-06-27T07:54:27.149Z" level=info msg="sub-process exited" argo=true error="<nil>"
retry-script-qztfz-retry-script-593990958: Error: exit with non-zero code. exit-code: 3, error:<nil>
retry-script-qztfz-retry-script-593990958: exit with non-zero code. exit-code: 3, error:<nil>
retry-script-qztfz-retry-script-4284816875: time="2023-06-27T07:55:32.695Z" level=info msg="sub-process exited" argo=true error="<nil>"
retry-script-qztfz-retry-script-4284816875: Error: exit with non-zero code. exit-code: 3, error:<nil>
retry-script-qztfz-retry-script-4284816875: exit with non-zero code. exit-code: 3, error:<nil>

Version

latest

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

# Only retry if the retryStrategy.expression condition is satisfied. In this example, retries will be made until a pod has
 # exit code 2 or the limit of 10 is reached, whichever happens first.
 apiVersion: argoproj.io/v1alpha1
 kind: Workflow
 metadata:
   generateName: retry-script-
 spec:
   entrypoint: main
   templates:
     - name: main
       steps:
         - - name: retry
             template: retry-script
     - name: retry-script
       retryStrategy:
         limit: "5"
         # Only continue retrying if the last exit code is greater than 1
         expression: "asInt(lastRetry.exitCode) > 1"
       nodeSelector:
         kubernetes.io/os: windows
       container:
         image: mcr.microsoft.com/windows-cssc/python3.7.2windows:ltsc2019
         command: ["python", "-c", "import random; import sys; exit_code = random.choice([1, 2, 3]);sys.exit(exit_code)"]

Logs from the workflow controller

time="2023-06-27T07:49:45.674Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:45.677Z" level=info msg="Updated phase  -> Running" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:45.678Z" level=info msg="Steps node retry-script-qztfz initialized Running" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:45.678Z" level=info msg="StepGroup node retry-script-qztfz-1518502507 initialized Running" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:45.678Z" level=info msg="Retry node retry-script-qztfz-2233819039 initialized Running" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:45.678Z" level=info msg="Pod node retry-script-qztfz-2876335522 initialized Pending" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:45.689Z" level=info msg="Created pod: retry-script-qztfz[0].retry(0) (retry-script-qztfz-retry-script-2876335522)" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:45.689Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:45.689Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:45.689Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:45.697Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45932283 workflow=retry-script-qztfz
time="2023-06-27T07:49:55.689Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:55.689Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:49:55.689Z" level=info msg="node changed" namespace=workflows new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=retry-script-qztfz-2876335522 old.message= old.phase=Pending old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:49:55.690Z" level=info msg="node retry-script-qztfz-2233819039 phase Running -> Pending" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:55.690Z" level=info msg="node retry-script-qztfz-2233819039 message: retryStrategy.expression evaluated to false" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:55.690Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:55.690Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:55.690Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:49:55.699Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45932378 workflow=retry-script-qztfz
time="2023-06-27T07:50:20.086Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:20.086Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:50:20.086Z" level=info msg="node changed" namespace=workflows new.message= new.phase=Running new.progress=0/1 nodeID=retry-script-qztfz-2876335522 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:50:20.087Z" level=info msg="node retry-script-qztfz-2233819039 phase Pending -> Running" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:20.087Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:20.087Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:20.087Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:20.095Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45932593 workflow=retry-script-qztfz
time="2023-06-27T07:50:30.096Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:30.096Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:50:30.096Z" level=info msg="node unchanged" namespace=workflows nodeID=retry-script-qztfz-2876335522 workflow=retry-script-qztfz
time="2023-06-27T07:50:30.097Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:30.097Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:30.097Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:50.197Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:50.198Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:50:50.198Z" level=info msg="Pod failed: Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>" displayName="retry(0)" namespace=workflows pod=retry-script-qztfz-retry-script-2876335522 templateName=retry-script workflow=retry-script-qztfz
time="2023-06-27T07:50:50.198Z" level=info msg="node changed" namespace=workflows new.message="Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>" new.phase=Failed new.progress=0/1 nodeID=retry-script-qztfz-2876335522 old.message= old.phase=Running old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:50:50.198Z" level=info msg="1 child nodes of retry-script-qztfz[0].retry failed. Trying again..." namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:50.198Z" level=info msg="Pod node retry-script-qztfz-2272194143 initialized Pending" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:50.208Z" level=info msg="Created pod: retry-script-qztfz[0].retry(1) (retry-script-qztfz-retry-script-2272194143)" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:50.208Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:50.208Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:50.208Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:50:50.216Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45932835 workflow=retry-script-qztfz
time="2023-06-27T07:50:50.222Z" level=info msg="cleaning up pod" action=labelPodCompleted key=workflows/retry-script-qztfz-retry-script-2876335522/labelPodCompleted
time="2023-06-27T07:51:00.210Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:00.210Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:51:00.210Z" level=info msg="node changed" namespace=workflows new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=retry-script-qztfz-2272194143 old.message= old.phase=Pending old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:51:00.211Z" level=info msg="node retry-script-qztfz-2233819039 phase Running -> Pending" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:00.211Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:00.211Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:00.211Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:00.218Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45932930 workflow=retry-script-qztfz
time="2023-06-27T07:51:25.349Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:25.349Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:51:25.349Z" level=info msg="node changed" namespace=workflows new.message= new.phase=Running new.progress=0/1 nodeID=retry-script-qztfz-2272194143 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:51:25.350Z" level=info msg="node retry-script-qztfz-2233819039 phase Pending -> Running" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:25.350Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:25.350Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:25.350Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:25.359Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45933134 workflow=retry-script-qztfz
time="2023-06-27T07:51:35.359Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:35.360Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:51:35.360Z" level=info msg="node unchanged" namespace=workflows nodeID=retry-script-qztfz-2272194143 workflow=retry-script-qztfz
time="2023-06-27T07:51:35.360Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:35.360Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:35.360Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:55.490Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:55.490Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:51:55.490Z" level=info msg="Pod failed: Error (exit code 64): exit with non-zero code. exit-code: 1, error:<nil>" displayName="retry(1)" namespace=workflows pod=retry-script-qztfz-retry-script-2272194143 templateName=retry-script workflow=retry-script-qztfz
time="2023-06-27T07:51:55.491Z" level=info msg="node changed" namespace=workflows new.message="Error (exit code 64): exit with non-zero code. exit-code: 1, error:<nil>" new.phase=Failed new.progress=0/1 nodeID=retry-script-qztfz-2272194143 old.message= old.phase=Running old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:51:55.491Z" level=info msg="2 child nodes of retry-script-qztfz[0].retry failed. Trying again..." namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:55.491Z" level=info msg="Pod node retry-script-qztfz-125203364 initialized Pending" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:55.503Z" level=info msg="Created pod: retry-script-qztfz[0].retry(2) (retry-script-qztfz-retry-script-125203364)" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:55.503Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:55.503Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:55.503Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:51:55.512Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45933367 workflow=retry-script-qztfz
time="2023-06-27T07:51:55.518Z" level=info msg="cleaning up pod" action=labelPodCompleted key=workflows/retry-script-qztfz-retry-script-2272194143/labelPodCompleted
time="2023-06-27T07:52:05.503Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:52:05.503Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:52:05.503Z" level=info msg="node changed" namespace=workflows new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=retry-script-qztfz-125203364 old.message= old.phase=Pending old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:52:05.504Z" level=info msg="node retry-script-qztfz-2233819039 phase Running -> Pending" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:52:05.504Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:52:05.504Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:52:05.504Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:52:05.512Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45933457 workflow=retry-script-qztfz
time="2023-06-27T07:52:30.633Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:52:30.633Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:52:30.633Z" level=info msg="node changed" namespace=workflows new.message= new.phase=Running new.progress=0/1 nodeID=retry-script-qztfz-125203364 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:52:30.634Z" level=info msg="node retry-script-qztfz-2233819039 phase Pending -> Running" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:52:30.634Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:52:30.634Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:52:30.634Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:52:30.641Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45933649 workflow=retry-script-qztfz
time="2023-06-27T07:53:00.745Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:00.745Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:53:00.746Z" level=info msg="Pod failed: Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>" displayName="retry(2)" namespace=workflows pod=retry-script-qztfz-retry-script-125203364 templateName=retry-script workflow=retry-script-qztfz
time="2023-06-27T07:53:00.746Z" level=info msg="node changed" namespace=workflows new.message="Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>" new.phase=Failed new.progress=0/1 nodeID=retry-script-qztfz-125203364 old.message= old.phase=Running old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:53:00.746Z" level=info msg="3 child nodes of retry-script-qztfz[0].retry failed. Trying again..." namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:00.746Z" level=info msg="Pod node retry-script-qztfz-594829601 initialized Pending" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:00.757Z" level=info msg="Created pod: retry-script-qztfz[0].retry(3) (retry-script-qztfz-retry-script-594829601)" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:00.757Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:00.757Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:00.757Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:00.768Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45933880 workflow=retry-script-qztfz
time="2023-06-27T07:53:00.775Z" level=info msg="cleaning up pod" action=labelPodCompleted key=workflows/retry-script-qztfz-retry-script-125203364/labelPodCompleted
time="2023-06-27T07:53:10.759Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:10.759Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:53:10.759Z" level=info msg="node changed" namespace=workflows new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=retry-script-qztfz-594829601 old.message= old.phase=Pending old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:53:10.760Z" level=info msg="node retry-script-qztfz-2233819039 phase Running -> Pending" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:10.760Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:10.760Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:10.760Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:10.769Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45933968 workflow=retry-script-qztfz
time="2023-06-27T07:53:32.877Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:32.877Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:53:32.878Z" level=info msg="node changed" namespace=workflows new.message= new.phase=Running new.progress=0/1 nodeID=retry-script-qztfz-594829601 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:53:32.878Z" level=info msg="node retry-script-qztfz-2233819039 phase Pending -> Running" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:32.878Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:32.878Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:32.878Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:53:32.887Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45934143 workflow=retry-script-qztfz
time="2023-06-27T07:54:03.021Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:03.021Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:54:03.022Z" level=info msg="Pod failed: Error (exit code 64): exit with non-zero code. exit-code: 1, error:<nil>" displayName="retry(3)" namespace=workflows pod=retry-script-qztfz-retry-script-594829601 templateName=retry-script workflow=retry-script-qztfz
time="2023-06-27T07:54:03.022Z" level=info msg="node changed" namespace=workflows new.message="Error (exit code 64): exit with non-zero code. exit-code: 1, error:<nil>" new.phase=Failed new.progress=0/1 nodeID=retry-script-qztfz-594829601 old.message= old.phase=Running old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:54:03.022Z" level=info msg="4 child nodes of retry-script-qztfz[0].retry failed. Trying again..." namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:03.022Z" level=info msg="Pod node retry-script-qztfz-593990958 initialized Pending" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:03.034Z" level=info msg="Created pod: retry-script-qztfz[0].retry(4) (retry-script-qztfz-retry-script-593990958)" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:03.034Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:03.034Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:03.034Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:03.044Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45934385 workflow=retry-script-qztfz
time="2023-06-27T07:54:03.050Z" level=info msg="cleaning up pod" action=labelPodCompleted key=workflows/retry-script-qztfz-retry-script-594829601/labelPodCompleted
time="2023-06-27T07:54:13.037Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:13.038Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:54:13.038Z" level=info msg="node changed" namespace=workflows new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=retry-script-qztfz-593990958 old.message= old.phase=Pending old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:54:13.038Z" level=info msg="node retry-script-qztfz-2233819039 phase Running -> Pending" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:13.038Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:13.038Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:13.038Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:13.047Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45934468 workflow=retry-script-qztfz
time="2023-06-27T07:54:23.048Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:23.048Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:54:23.048Z" level=info msg="node unchanged" namespace=workflows nodeID=retry-script-qztfz-593990958 workflow=retry-script-qztfz
time="2023-06-27T07:54:23.048Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:23.048Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:23.048Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:37.391Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:37.392Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:54:37.392Z" level=info msg="node changed" namespace=workflows new.message= new.phase=Running new.progress=0/1 nodeID=retry-script-qztfz-593990958 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:54:37.392Z" level=info msg="node retry-script-qztfz-2233819039 phase Pending -> Running" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:37.392Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:37.392Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:37.392Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:54:37.401Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45934662 workflow=retry-script-qztfz
time="2023-06-27T07:55:07.701Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:07.701Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:55:07.701Z" level=info msg="Pod failed: Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>" displayName="retry(4)" namespace=workflows pod=retry-script-qztfz-retry-script-593990958 templateName=retry-script workflow=retry-script-qztfz
time="2023-06-27T07:55:07.701Z" level=info msg="node changed" namespace=workflows new.message="Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>" new.phase=Failed new.progress=0/1 nodeID=retry-script-qztfz-593990958 old.message= old.phase=Running old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:55:07.702Z" level=info msg="5 child nodes of retry-script-qztfz[0].retry failed. Trying again..." namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:07.702Z" level=info msg="Pod node retry-script-qztfz-4284816875 initialized Pending" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:07.722Z" level=info msg="Created pod: retry-script-qztfz[0].retry(5) (retry-script-qztfz-retry-script-4284816875)" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:07.722Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:07.722Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:07.722Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:07.735Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45934901 workflow=retry-script-qztfz
time="2023-06-27T07:55:07.742Z" level=info msg="cleaning up pod" action=labelPodCompleted key=workflows/retry-script-qztfz-retry-script-593990958/labelPodCompleted
time="2023-06-27T07:55:17.725Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:17.725Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:55:17.726Z" level=info msg="node changed" namespace=workflows new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=retry-script-qztfz-4284816875 old.message= old.phase=Pending old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:55:17.726Z" level=info msg="node retry-script-qztfz-2233819039 phase Running -> Pending" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:17.726Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:17.727Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:17.727Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:17.737Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45934992 workflow=retry-script-qztfz
time="2023-06-27T07:55:42.896Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:42.896Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:55:42.896Z" level=info msg="node changed" namespace=workflows new.message= new.phase=Running new.progress=0/1 nodeID=retry-script-qztfz-4284816875 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:55:42.897Z" level=info msg="node retry-script-qztfz-2233819039 phase Pending -> Running" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:42.897Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:42.897Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:42.897Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:42.906Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=45935180 workflow=retry-script-qztfz
time="2023-06-27T07:55:52.906Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:52.906Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:55:52.906Z" level=info msg="node unchanged" namespace=workflows nodeID=retry-script-qztfz-4284816875 workflow=retry-script-qztfz
time="2023-06-27T07:55:52.906Z" level=info msg="Workflow step group node retry-script-qztfz-1518502507 not yet completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:52.906Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:55:52.906Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="Processing workflow" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="Pod failed: Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>" displayName="retry(5)" namespace=workflows pod=retry-script-qztfz-retry-script-4284816875 templateName=retry-script workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="node changed" namespace=workflows new.message="Error (exit code 64): exit with non-zero code. exit-code: 3, error:<nil>" new.phase=Failed new.progress=0/1 nodeID=retry-script-qztfz-4284816875 old.message= old.phase=Running old.progress=0/1 workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="No more retries left. Failing..." namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="node retry-script-qztfz-2233819039 phase Running -> Failed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="node retry-script-qztfz-2233819039 message: No more retries left" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="node retry-script-qztfz-2233819039 finished: 2023-06-27 07:56:13.006820757 +0000 UTC" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="Step group node retry-script-qztfz-1518502507 deemed failed: child 'retry-script-qztfz-2233819039' failed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="node retry-script-qztfz-1518502507 phase Running -> Failed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="node retry-script-qztfz-1518502507 message: child 'retry-script-qztfz-2233819039' failed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="node retry-script-qztfz-1518502507 finished: 2023-06-27 07:56:13.006882112 +0000 UTC" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="step group retry-script-qztfz-1518502507 was unsuccessful: child 'retry-script-qztfz-2233819039' failed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="Outbound nodes of retry-script-qztfz-2233819039 is [retry-script-qztfz-4284816875]" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="Outbound nodes of retry-script-qztfz is [retry-script-qztfz-4284816875]" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="node retry-script-qztfz phase Running -> Failed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="node retry-script-qztfz message: child 'retry-script-qztfz-2233819039' failed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="node retry-script-qztfz finished: 2023-06-27 07:56:13.006957524 +0000 UTC" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.006Z" level=info msg="Checking daemoned children of retry-script-qztfz" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.007Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.007Z" level=info msg=reconcileAgentPod namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.007Z" level=info msg="Updated phase Running -> Failed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.007Z" level=info msg="Updated message  -> child 'retry-script-qztfz-2233819039' failed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.007Z" level=info msg="Marking workflow completed" namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.007Z" level=info msg="Checking daemoned children of " namespace=workflows workflow=retry-script-qztfz
time="2023-06-27T07:56:13.012Z" level=info msg="cleaning up pod" action=deletePod key=workflows/retry-script-qztfz-1340600742-agent/deletePod
time="2023-06-27T07:56:13.016Z" level=info msg="Workflow update successful" namespace=workflows phase=Failed resourceVersion=45935415 workflow=retry-script-qztfz
time="2023-06-27T07:56:13.026Z" level=info msg="cleaning up pod" action=labelPodCompleted key=workflows/retry-script-qztfz-retry-script-4284816875/labelPodCompleted

Logs from in your workflow's wait container

time="2023-06-27T07:50:06.346Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2023-06-27T07:50:06.347Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=render-cust-helio podName=retry-script-qztfz-retry-script-2876335522 template="{\"name\":\"retry-script\",\"inputs\":{},\"outputs\":{},\"nodeSelector\":{\"kubernetes.io/os\":\"windows\",\"node.helio.exchange/3dsmax-base-version\":\"v1.2.0\",\"node.helio.exchange/os\":\"windows1809\",\"node.helio.exchange/render-engine-max-2023\":\"true\"},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"eu.gcr.io/k8s-helio-development/rendering/windows/max2023-vray6-wrapped:1.30.0\",\"command\":[\"python\",\"-c\",\"import random; import sys; exit_code = random.choice([1, 2, 3]);sys.exit(exit_code)\"],\"resources\":{}},\"retryStrategy\":{\"limit\":\"5\",\"expression\":\"asInt(lastRetry.exitCode) \\u003e 1\"},\"tolerations\":[{\"key\":\"node.helio.exchange/role\",\"value\":\"render\",\"effect\":\"NoSchedule\"}]}" version="&Version{Version:v3.4.5+1253f44.dirty,BuildDate:2023-02-07T12:41:30Z,GitCommit:1253f443baa8ad1610d2e62ec26ecdc85fe1b837,GitTag:v3.4.5,GitTreeState:dirty,GoVersion:go1.18,Compiler:gc,Platform:windows/amd64,}"
time="2023-06-27T07:50:06.347Z" level=info msg="Starting deadline monitor"
time="2023-06-27T07:50:10.349Z" level=info msg="Main container completed" error="<nil>"
time="2023-06-27T07:50:10.349Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2023-06-27T07:50:10.349Z" level=info msg="No output parameters"
time="2023-06-27T07:50:10.349Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2023-06-27T07:50:10.349Z" level=info msg="No output artifacts"
time="2023-06-27T07:50:10.349Z" level=info msg="Deadline monitor stopped"
time="2023-06-27T07:50:10.349Z" level=info msg="Alloc=6970 TotalAlloc=11846 Sys=22176 NumGC=3 Goroutines=5"
@juliev0 juliev0 added the P3 Low priority label Jun 29, 2023
cbuchli added a commit to helio/argo-workflows that referenced this issue Oct 4, 2023
dpadhiar pushed a commit to dpadhiar/argo-workflows that referenced this issue May 9, 2024
@agilgur5 agilgur5 added the area/windows Windows Container support label Jun 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/executor area/windows Windows Container support P3 Low priority type/bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants