Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tkn logs is not displaying tasks defined in the pipeline with 'retries: X' #2208

Closed
dmitry-mightydevops opened this issue Jan 8, 2024 · 0 comments · Fixed by #2251
Closed

tkn logs is not displaying tasks defined in the pipeline with 'retries: X' #2208

dmitry-mightydevops opened this issue Jan 8, 2024 · 0 comments · Fixed by #2251
Assignees
@piyush-garg
Labels
kind/bug Categorizes issue or PR as related to a bug.
Milestone
0.36.0

Comments

@dmitry-mightydevops
Copy link

dmitry-mightydevops commented Jan 8, 2024 •
edited
Loading

Versions and Operating System

  • Kubernetes version: 1.27 (aws eks)
  • Linux only nodes (AMD64)
  • Tekton Pipeline version:
➜ tkn version                                 
Client version: 0.33.0
Pipeline version: v0.54.0
Triggers version: v0.25.3
Dashboard version: v0.42.0

Expected Behavior

tkn -n ci  pipelinerun logs -f NAME

should display logs of tasks that are defined with "retries: N".

Actual Behavior

The output is completely ignored, yet you can get it via kubectl logs -f PODNAME associated with the step
As soon as I removed the attribute 'retries' - I see log output normally

Steps to Reproduce the Problem

➜ which tknlo             
tknlo='tkn -n ci  pipelinerun logs -f'
	/usr/bin/tkn

NO RETRIES is set for the task in the pipeline

tknlo output

[build-kaniko : build] Successfully installed Flask-2.0.2 Jinja2-3.0.2 MarkupSafe-2.0.1 Werkzeug-2.0.2 click-8.0.3 itsdangerous-2.0.1
[build-kaniko : build] WARNING: You are using pip version 21.0.1; however, version 23.3.2 is available.
[build-kaniko : build] You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
[build-kaniko : build] INFO[0024] Taking snapshot of full filesystem...        
[build-kaniko : build] INFO[0026] COPY . .                                     
[build-kaniko : build] INFO[0026] Taking snapshot of files...                  
[build-kaniko : build] INFO[0026] CMD [ "python3", "app.py"]                   
[build-kaniko : build] INFO[0026] Pushing image to 111.dkr.ecr.us-east-1.amazonaws.com/111/prod/backend:latest13 
[build-kaniko : build] INFO[0027] Pushed 111.dkr.ecr.us-east-1.amazonaws.com/111/prod/backend@sha256:f779d655b5a090451e922a10e5c29f161c3df55d41263b32bab4592d93a14ddc 
[build-kaniko : build] INFO[0028] Pushing image to 111.dkr.ecr.us-east-1.amazonaws.com/111/prod/backend:kba13e 
[build-kaniko : build] INFO[0028] Pushed 111.dkr.ecr.us-east-1.amazonaws.com/111/prod/backend@sha256:f779d655b5a090451e922a10e5c29f161c3df55d41263b32bab4592d93a14ddc 

[build-kaniko : results] 111.dkr.ecr.us-east-1.amazonaws.com/111/prod/backend:kba13e

[gitops : clone-kubernetes-repo] Cloning into '/workspace/source/gitops/kubernetes-repo'...
[gitops : clone-kubernetes-repo] branch 'feature/add-prod' set up to track 'origin/feature/add-prod'.
[gitops : clone-kubernetes-repo] Switched to a new branch 'feature/add-prod'
[gitops : clone-kubernetes-repo] HEAD is now at dbd94df gitops: new image sha256:2c9ea6f403f41886fa04e18d1f823eff2a1eb84065e7e7378902b01f512344a5 compiled by tekton in prod env

[gitops : update-image] 
[gitops : update-image] ===> PATCH APPS/PROD/VALUES.YAML FILE WITH IMAGE_DIGEST
[gitops : update-image] 
[gitops : update-image] patching file apps/prod/values.yaml
[gitops : update-image] 
[gitops : update-image] ===> GITOPS DIFF
[gitops : update-image] 
[gitops : update-image] diff --git a/apps/prod/values.yaml b/apps/prod/values.yaml
[gitops : update-image] index d8f1fe1..a7b5fe4 100644
[gitops : update-image] --- a/apps/prod/values.yaml
[gitops : update-image] +++ b/apps/prod/values.yaml
[gitops : update-image] @@ -22,7 +22,7 @@ apps:
[gitops : update-image]      deployment:
[gitops : update-image]        image:
[gitops : update-image]          repository: 111.dkr.ecr.us-east-1.amazonaws.com/111/prod/backend
[gitops : update-image] -        digest: sha256:2c9ea6f403f41886fa04e18d1f823eff2a1eb84065e7e7378902b01f512344a5
[gitops : update-image] +        digest: sha256:f779d655b5a090451e922a10e5c29f161c3df55d41263b32bab4592d93a14ddc
[gitops : update-image]        command:
[gitops : update-image]          - bash
[gitops : update-image]          - -c
[gitops : update-image] 
[gitops : update-image] ===> GITOPS PUSH TO HTTPS://GITHUB.COM/SARITASA-NEST/USUMMIT-KUBERNETES-AWS
[gitops : update-image] 
[gitops : update-image] [feature/add-prod 9e6e40f] gitops: new image sha256:f779d655b5a090451e922a10e5c29f161c3df55d41263b32bab4592d93a14ddc compiled by tekton in prod env
[gitops : update-image]  1 file changed, 1 insertion(+), 1 deletion(-)
[gitops : update-image] To https://github.com/org/111-kubernetes-aws
[gitops : update-image]    dbd94df..9e6e40f  feature/add-prod -> feature/add-prod

[deploy : execute] 
[deploy : execute] ===> LOGIN TO ARGOCD
[deploy : execute] 
[deploy : execute] 'admin:login' logged in successfully
[deploy : execute] Context 'argocd-server.argo-cd.svc' updated
[deploy : execute] 
[deploy : execute] ===> GET ARGOCD VERSION
[deploy : execute]

history of pods

buildpack-pipeline-git-clone-pod               0/1     Init:0/2            0          12s
buildpack-pipeline-git-clone-pod               0/1     Init:1/2            0          13s
buildpack-pipeline-git-clone-pod               0/1     PodInitializing     0          14s
buildpack-pipeline-git-clone-pod               1/1     Running             0          15s
buildpack-pipeline-git-clone-pod               1/1     Running             0          15s
buildpack-pipeline-git-clone-pod               0/1     Completed           0          22s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Pending             0          0s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Pending             0          0s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Init:0/2            0          0s
buildpack-pipeline-git-clone-pod               0/1     Completed           0          23s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Init:1/2            0          2s
buildpack-pipeline-git-clone-pod               0/1     Completed           0          24s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     PodInitializing     0          3s
buildpack-pipeline-build-kaniko-env-vars-pod   1/1     Running             0          4s
buildpack-pipeline-build-kaniko-env-vars-pod   1/1     Running             0          4s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Completed           0          5s
buildpack-pipeline-build-kaniko-pod            0/2     Pending             0          0s
buildpack-pipeline-build-kaniko-pod            0/2     Pending             0          0s
buildpack-pipeline-build-kaniko-pod            0/2     Init:0/3            0          0s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Completed           0          6s
buildpack-pipeline-build-kaniko-pod            0/2     Init:0/3            0          2s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Completed           0          7s
buildpack-pipeline-build-kaniko-pod            0/2     Init:1/3            0          3s
buildpack-pipeline-build-kaniko-pod            0/2     Init:2/3            0          4s
buildpack-pipeline-build-kaniko-pod            0/2     PodInitializing     0          5s
buildpack-pipeline-build-kaniko-pod            2/2     Running             0          6s
buildpack-pipeline-build-kaniko-pod            2/2     Running             0          6s
buildpack-pipeline-build-kaniko-pod            1/2     NotReady            0          36s
buildpack-pipeline-build-kaniko-pod            0/2     Completed           0          37s
buildpack-pipeline-gitops-pod                  0/2     Pending             0          0s
buildpack-pipeline-gitops-pod                  0/2     Pending             0          0s
buildpack-pipeline-gitops-pod                  0/2     Init:0/3            0          0s
buildpack-pipeline-build-kaniko-pod            0/2     Completed           0          38s
buildpack-pipeline-gitops-pod                  0/2     Init:1/3            0          2s
buildpack-pipeline-build-kaniko-pod            0/2     Completed           0          39s
buildpack-pipeline-gitops-pod                  0/2     Init:2/3            0          3s
buildpack-pipeline-gitops-pod                  0/2     PodInitializing     0          4s
buildpack-pipeline-gitops-pod                  2/2     Running             0          5s
buildpack-pipeline-gitops-pod                  2/2     Running             0          5s
buildpack-pipeline-gitops-pod                  1/2     NotReady            0          8s
buildpack-pipeline-gitops-pod                  0/2     Completed           0          10s
buildpack-pipeline-deploy-pod                  0/1     Pending             0          0s

RETRIES is set for the task in the pipeline

when added retries to pipeline task:

- name: gitops
  retries: 3    <-----------THIS 
  when:
    - input: $(params.workflow.gitops)
      operator: in
      values: ['true']
  taskRef:
    name: gitops

tknlo output

tkn logs do not output the task output.

So you can see that after build-kaniko we immediately see the deploy task output, and the gitops task is completely ignorer, however it has finished successfully and no restart were necessary.

build-kaniko : build] Collecting Jinja2==3.0.2
[build-kaniko : build]   Downloading Jinja2-3.0.2-py3-none-any.whl (133 kB)
[build-kaniko : build] Collecting MarkupSafe==2.0.1
[build-kaniko : build]   Downloading MarkupSafe-2.0.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (30 kB)
[build-kaniko : build] Collecting Werkzeug==2.0.2
[build-kaniko : build]   Downloading Werkzeug-2.0.2-py3-none-any.whl (288 kB)
[build-kaniko : build] Installing collected packages: MarkupSafe, Werkzeug, Jinja2, itsdangerous, click, Flask
[build-kaniko : build] Successfully installed Flask-2.0.2 Jinja2-3.0.2 MarkupSafe-2.0.1 Werkzeug-2.0.2 click-8.0.3 itsdangerous-2.0.1
[build-kaniko : build] WARNING: You are using pip version 21.0.1; however, version 23.3.2 is available.
[build-kaniko : build] You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
[build-kaniko : build] INFO[0025] Taking snapshot of full filesystem...        
[build-kaniko : build] INFO[0027] COPY . .                                     
[build-kaniko : build] INFO[0027] Taking snapshot of files...                  
[build-kaniko : build] INFO[0027] CMD [ "python3", "app.py"]                   
[build-kaniko : build] INFO[0027] Pushing image to 111.dkr.ecr.us-east-1.amazonaws.com/111/prod/backend:latest14 
[build-kaniko : build] INFO[0029] Pushed 111.dkr.ecr.us-east-1.amazonaws.com/111/prod/backend@sha256:f5ea1f17347f0d3867f3c6684b0033daab029936b94436eed18eec8ff90d8f11 
[build-kaniko : build] INFO[0029] Pushing image to 111.dkr.ecr.us-east-1.amazonaws.com/111/prod/backend:kba14e 
[build-kaniko : build] INFO[0029] Pushed 111.dkr.ecr.us-east-1.amazonaws.com/111/prod/backend@sha256:f5ea1f17347f0d3867f3c6684b0033daab029936b94436eed18eec8ff90d8f11 

[build-kaniko : results] 111.dkr.ecr.us-east-1.amazonaws.com/111/prod/backend:kba14e

[deploy : execute] 
[deploy : execute] ===> LOGIN TO ARGOCD
[deploy : execute] 
[deploy : execute] 'admin:login' logged in successfully
[deploy : execute] Context 'argocd-server.argo-cd.svc' updated
[deploy : execute] 
[deploy : execute] ===> GET ARGOCD VERSION

history of pods

✗  kgpo -w
NAME                                  READY   STATUS      RESTARTS   AGE
affinity-assistant-a5f6edee78-0       0/1     Pending     0          1s
buildpack-pipeline-git-clone-pod      0/1     Pending     0          1s
buildpack-pipeline-gitops-nx6gm-pod   0/2     Completed   0          62m
affinity-assistant-a5f6edee78-0       0/1     Pending     0          5s
affinity-assistant-a5f6edee78-0       0/1     ContainerCreating   0          5s
buildpack-pipeline-git-clone-pod      0/1     Pending             0          5s
buildpack-pipeline-git-clone-pod      0/1     Init:0/2            0          5s
affinity-assistant-a5f6edee78-0       1/1     Running             0          11s
buildpack-pipeline-git-clone-pod      0/1     Init:1/2            0          14s
buildpack-pipeline-git-clone-pod      0/1     PodInitializing     0          15s
buildpack-pipeline-git-clone-pod      1/1     Running             0          16s
buildpack-pipeline-git-clone-pod      1/1     Running             0          16s
buildpack-pipeline-git-clone-pod      0/1     Completed           0          22s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Pending             0          0s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Pending             0          0s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Init:0/2            0          0s
buildpack-pipeline-git-clone-pod               0/1     Completed           0          23s
buildpack-pipeline-git-clone-pod               0/1     Completed           0          24s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Init:1/2            0          2s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     PodInitializing     0          3s
buildpack-pipeline-build-kaniko-env-vars-pod   1/1     Running             0          4s
buildpack-pipeline-build-kaniko-env-vars-pod   1/1     Running             0          4s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Completed           0          7s
buildpack-pipeline-build-kaniko-pod            0/2     Pending             0          0s
buildpack-pipeline-build-kaniko-pod            0/2     Pending             0          1s
buildpack-pipeline-build-kaniko-pod            0/2     Init:0/3            0          1s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Completed           0          8s
buildpack-pipeline-build-kaniko-env-vars-pod   0/1     Completed           0          9s
buildpack-pipeline-build-kaniko-pod            0/2     Init:1/3            0          2s
buildpack-pipeline-build-kaniko-pod            0/2     Init:2/3            0          3s
buildpack-pipeline-build-kaniko-pod            0/2     PodInitializing     0          4s
buildpack-pipeline-build-kaniko-pod            2/2     Running             0          5s
buildpack-pipeline-build-kaniko-pod            2/2     Running             0          5s
buildpack-pipeline-build-kaniko-pod            1/2     NotReady            0          36s
buildpack-pipeline-build-kaniko-pod            0/2     Completed           0          37s
buildpack-pipeline-gitops-pod                  0/2     Pending             0          0s
buildpack-pipeline-gitops-pod                  0/2     Pending             0          0s
buildpack-pipeline-gitops-pod                  0/2     Init:0/3            0          0s
buildpack-pipeline-build-kaniko-pod            0/2     Completed           0          39s
buildpack-pipeline-gitops-pod                  0/2     Init:1/3            0          1s
buildpack-pipeline-build-kaniko-pod            0/2     Completed           0          39s
buildpack-pipeline-gitops-pod                  0/2     Init:2/3            0          2s
buildpack-pipeline-gitops-pod                  0/2     PodInitializing     0          4s
buildpack-pipeline-gitops-pod                  2/2     Running             0          5s
buildpack-pipeline-gitops-pod                  2/2     Running             0          5s
buildpack-pipeline-gitops-pod                  1/2     NotReady            0          6s
buildpack-pipeline-gitops-pod                  0/2     Completed           0          8s
buildpack-pipeline-deploy-pod                  0/1     Pending             0          0s
buildpack-pipeline-deploy-pod                  0/1     Pending             0          0s
buildpack-pipeline-deploy-pod                  0/1     Init:0/3            0          0s
buildpack-pipeline-gitops-pod                  0/2     Completed           0          10s
buildpack-pipeline-deploy-pod                  0/1     Init:1/3            0          1s
buildpack-pipeline-gitops-pod                  0/2     Completed           0          11s
buildpack-pipeline-deploy-pod                  0/1     Init:2/3            0          3s
buildpack-pipeline-deploy-pod                  0/1     PodInitializing     0          4s
buildpack-pipeline-deploy-pod                  1/1     Running             0          5s
buildpack-pipeline-deploy-pod                  1/1     Running             0          5s
buildpack-pipeline-deploy-pod                  0/1     Completed           0          17s
buildpack-pipeline-notify-pod                  0/7     Pending             0          0s
buildpack-pipeline-notify-pod                  0/7     Pending             0          0s
buildpack-pipeline-notify-pod                  0/7     Init:0/3            0          0s
buildpack-pipeline-deploy-pod                  0/1     Completed           0          18s
buildpack-pipeline-deploy-pod                  0/1     Completed           0          19s
buildpack-pipeline-notify-pod                  0/7     Init:1/3            0          2s
buildpack-pipeline-notify-pod                  0/7     Init:2/3            0          3s
buildpack-pipeline-notify-pod                  0/7     PodInitializing     0          4s
buildpack-pipeline-notify-pod                  7/7     Running             0          6s
buildpack-pipeline-notify-pod                  7/7     Running             0          6s
buildpack-pipeline-notify-pod                  6/7     NotReady            0          9s
buildpack-pipeline-notify-pod                  2/7     NotReady            0          10s
buildpack-pipeline-notify-pod                  1/7     NotReady            0          11s
buildpack-pipeline-notify-pod                  0/7     Completed           0          12s
affinity-assistant-a5f6edee78-0                1/1     Terminating         0          106s
affinity-assistant-a5f6edee78-0                0/1     Terminating         0          106s
affinity-assistant-a5f6edee78-0                0/1     Terminating         0          107s
affinity-assistant-a5f6edee78-0                0/1     Terminating         0          107s
affinity-assistant-a5f6edee78-0                0/1     Terminating         0          107s
buildpack-pipeline-notify-pod                  0/7     Completed           0          13s

Additional Info
@dmitry-mightydevops dmitry-mightydevops added the kind/bug Categorizes issue or PR as related to a bug. label Jan 8, 2024
@piyush-garg piyush-garg self-assigned this Feb 21, 2024
piyush-garg added a commit to piyush-garg/cli that referenced this issue Feb 21, 2024
@piyush-garg
Fix logs command not showing logs
db1bcd7 
This will fix the logs command not showing logs
if there are retries and task has completed in first go

Failure and Completion of taskrun should not
depend on no. of retries. Also retries will
not scheduled be if task is done so add a check
for that.

Add e2e test for this scenarioi

Fix tektoncd#2208
@piyush-garg piyush-garg mentioned this issue Feb 21, 2024
Merged
4 tasks
piyush-garg added a commit to piyush-garg/cli that referenced this issue Feb 21, 2024
@piyush-garg
Fix logs command not showing logs
4bbaa44 
This will fix the logs command not showing logs
if there are retries and task has completed in first go

Failure and Completion of taskrun should not
depend on no. of retries. Also retries will
not scheduled be if task is done so add a check
for that.

Add e2e test for this scenarioi

Fix tektoncd#2208
@piyush-garg piyush-garg added this to the 0.36.0 milestone Feb 27, 2024
tekton-robot pushed a commit that referenced this issue Feb 27, 2024
@piyush-garg @tekton-robot
Fix logs command not showing logs
22440cd 
This will fix the logs command not showing logs
if there are retries and task has completed in first go

Failure and Completion of taskrun should not
depend on no. of retries. Also retries will
not scheduled be if task is done so add a check
for that.

Add e2e test for this scenarioi

Fix #2208
piyush-garg added a commit that referenced this issue Mar 25, 2024
@piyush-garg
Fix logs command not showing logs
aaa589a 
This will fix the logs command not showing logs
if there are retries and task has completed in first go

Failure and Completion of taskrun should not
depend on no. of retries. Also retries will
not scheduled be if task is done so add a check
for that.

Add e2e test for this scenarioi

Fix #2208
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@piyush-garg piyush-garg

Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
0.36.0
Development

Successfully merging a pull request may close this issue.

Fix logs command not showing logs piyush-garg/cli
2 participants
@piyush-garg @dmitry-mightydevops