Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crashing running "tkn pipeline logs --follow" #1080

Closed
chmouel opened this issue Jul 22, 2020 · 6 comments
Closed

Crashing running "tkn pipeline logs --follow" #1080

chmouel opened this issue Jul 22, 2020 · 6 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.
Milestone
0.11.0

Comments

@chmouel
Copy link
Member

chmouel commented Jul 22, 2020 •
edited
Loading

Versions and Operating System

  • Kubernetes version:

    Output of kubectl version:

    Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2-0-g52c56ce", GitCommit:"9933eb90790b36d153fcc55f8404724bb0929b96", GitTreeState:"clean", BuildDate:"2020-05-29T14:25:43Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"darwin/amd64"}

  • Tekton Pipeline version:

    Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

    nightly as 20200722

  • Operating System:

    OSX latest

Expected Behavior

working

Actual Behavior

% tkn p logs go-simple-uploader-pr --follow
[fetch : clone] + CHECKOUT_DIR=/workspace/output/
[fetch : clone] + '[[' false '==' true ]]
[fetch : clone] + test -z
[fetch : clone] + test -z
[fetch : clone] + test -z
[fetch : clone] + /ko-app/git-init -url https://github.com/chmouel/go-simple-uploader -revision 1e724c4f2e4bd26ef4feb497a609d7a1ff4cdb0e -refspec  -path /workspace/output/ '-sslVerify=true' '-submodules=true' -depth 1
[fetch : clone] {"level":"info","ts":1595414634.3231862,"caller":"git/git.go:139","msg":"Successfully cloned https://github.com/chmouel/go-simple-uploader @ 1e724c4f2e4bd26ef4feb497a609d7a1ff4cdb0e (grafted, HEAD) in path /workspace/output/"}
[fetch : clone] {"level":"info","ts":1595414634.3639627,"caller":"git/git.go:180","msg":"Successfully initialized and updated submodules in path /workspace/output/"}
[fetch : clone] + cd /workspace/output/
[fetch : clone] + git rev-parse HEAD
[fetch : clone] + tr -d '\n'
[fetch : clone] + RESULT_SHA=1e724c4f2e4bd26ef4feb497a609d7a1ff4cdb0e
[fetch : clone] + EXIT_CODE=0
[fetch : clone] + '[' 0 '!=' 0 ]
[fetch : clone] + echo -n 1e724c4f2e4bd26ef4feb497a609d7a1ff4cdb0e
E0722 12:44:45.984093   96052 runtime.go:78] Observed a panic: "send on closed channel" (send on closed channel)
goroutine 1932 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x21eca00, 0x2605bd0)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x21eca00, 0x2605bd0)
	/usr/local/opt/go/libexec/src/runtime/panic.go:967 +0x15d
github.com/tektoncd/cli/pkg/pods.(*Pod).watcher.func2(0x23b27c0, 0xc000501400, 0x23b27c0, 0xc000501c00)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/pkg/pods/pod.go:106 +0x48
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/controller.go:205
k8s.io/client-go/tools/cache.(*processorListener).run.func1.1(0xc0009fe1d8, 0x2, 0x1)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/shared_informer.go:656 +0x1e8
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0x0, 0xc000a36e18, 0x200c000729400, 0x0)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:292 +0x51
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/shared_informer.go:652 +0x79
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000266760)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000a36f60, 0xdf8475800, 0x0, 0x1, 0xc0004e0a20)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000976080)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/shared_informer.go:650 +0x9b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000226220, 0xc00075d7e0)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62
panic: send on closed channel [recovered]
	panic: send on closed channel

goroutine 1932 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x21eca00, 0x2605bd0)
	/usr/local/opt/go/libexec/src/runtime/panic.go:967 +0x15d
github.com/tektoncd/cli/pkg/pods.(*Pod).watcher.func2(0x23b27c0, 0xc000501400, 0x23b27c0, 0xc000501c00)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/pkg/pods/pod.go:106 +0x48
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/controller.go:205
k8s.io/client-go/tools/cache.(*processorListener).run.func1.1(0xc0009fe1d8, 0x2, 0x1)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/shared_informer.go:656 +0x1e8
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x989680, 0x3ff0000000000000, 0x3fb999999999999a, 0x5, 0x0, 0xc000517e18, 0x200c000729400, 0x0)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:292 +0x51
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/shared_informer.go:652 +0x79
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000266760)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000517f60, 0xdf8475800, 0x0, 0x1, 0xc0004e0a20)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000976080)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/shared_informer.go:650 +0x9b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000226220, 0xc00075d7e0)
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/Users/chmouel/GIT/go/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62
chmouel@TheBatman 🦇 .../go-simple-uploader/tekton OS4/NONE tekton
%

/kind bug

Steps to Reproduce the Problem

  1. run a tkn p logs pipeline --follow

perhaps this only occurs on my env and it's a random env error but we need to test if we can reproduce it first...

Additional Info

Latest version from trunk compiled as of the time this bug reported
@chmouel chmouel added the kind/bug Categorizes issue or PR as related to a bug. label Jul 22, 2020
@piyush-garg piyush-garg added this to the 0.11.0 milestone Jul 22, 2020
@chmouel chmouel changed the title Crashing running "tkn pipeline logs" Crashing running "tkn pipeline logs --follow" Jul 22, 2020
@pradeepitm12
Copy link
Contributor

closing: Tried with few example pipelines and was not able to reproduce,

@pradeepitm12
Copy link
Contributor

/close

@tekton-robot
Copy link
Contributor

@pradeepitm12: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@chmouel
Copy link
Member Author

chmouel commented Jul 23, 2020

Thanks for traxking this @pradeepitm12 i could nto reproduce either...

@sudo-bmitch sudo-bmitch mentioned this issue Jul 21, 2022
Closed
@keskad
Copy link

keskad commented Jun 21, 2023 •
edited
Loading 

13:38:53  [create-release-pr : create-merge-request] DEBUG:urllib3.connectionpool:http://xxxx.int:80 "PUT /api/v4/projects/yyy/merge_requests/zzz/merge HTTP/1.1" 200 None13:38:53  [create-release-pr : create-merge-request] INFO:root:Leaving a comment on the PR to inform users whats going on13:38:53  [create-release-pr : create-merge-request] DEBUG:urllib3.connectionpool:http://xxxxx.int:80 "POST /api/v4/projects/yyy/merge_requests/zzz/discussions HTTP/1.1" 201 77313:38:54  E0621 13:38:54.367349  272526 runtime.go:79] Observed a panic: "send on closed channel" (send on closed channel)13:38:54  goroutine 376 [running]:13:38:54  k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2c9b620?, 0x38cfb90})13:38:54      /workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x9913:38:54  k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x0?})13:38:54      /workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
13:38:54  panic({0x2c9b620, 0x38cfb90})

I see this panic second time recently. It appeared I think after upgrading Tekton Pipelines recently.
After recent upgrade I got two issues: OpenShift Pipelines stopped showing valid visualization (pipelines are always in "not started" state on the list, and in details page a pipeline has always failed all stages) and this one, where tkn panics second time.

Tekton Pipelines: v0.47.2
Tekton Triggers: v0.24.1
Pipelines as Code: 0.19.2
Tekton Chains: v0.16.0
Tekton Hub (tech-preview): v1.13.0
Tekton Results (tech-preview): v0.6.0

tkn CLI v0.30.0, now upgrading to latest and will be observing 🙂

@keskad
Copy link

keskad commented Jun 21, 2023 •
edited
Loading

Now I got it from my computer also:

E0621 12:41:34.300814      29 runtime.go:79] Observed a panic: "send on closed channel" (send on closed channel)
goroutine 382 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2c9b620?, 0x38cfb90})
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x0?})
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
panic({0x2c9b620, 0x38cfb90})
	/usr/local/go/src/runtime/panic.go:884 +0x212
github.com/tektoncd/cli/pkg/pipelinerun.(*Tracker).Monitor.func2({0x3173700?, 0xc001396480?})
	/workspace/src/github.com/tektoncd/cli/pkg/pipelinerun/tracker.go:101 +0x1af
github.com/tektoncd/cli/pkg/pipelinerun.(*Tracker).Monitor.func3({0x406878?, 0xc000c60480?}, {0x3173700?, 0xc001396480?})
	/workspace/src/github.com/tektoncd/cli/pkg/pipelinerun/tracker.go:111 +0x2d
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/controller.go:239
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/shared_informer.go:816 +0xf7
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x1c00032a640?)
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x46ee67?, {0x38dea60, 0xc001358000}, 0x1, 0xc0009ac000)
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x1b?, 0x3b9aca00, 0x0, 0xa0?, 0xc00032a788?)
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000206d80?)
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/shared_informer.go:812 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85
panic: send on closed channel [recovered]
	panic: send on closed channel

goroutine 382 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x0?})
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x2c9b620, 0x38cfb90})
	/usr/local/go/src/runtime/panic.go:884 +0x212
github.com/tektoncd/cli/pkg/pipelinerun.(*Tracker).Monitor.func2({0x3173700?, 0xc001396480?})
	/workspace/src/github.com/tektoncd/cli/pkg/pipelinerun/tracker.go:101 +0x1af
github.com/tektoncd/cli/pkg/pipelinerun.(*Tracker).Monitor.func3({0x406878?, 0xc000c60480?}, {0x3173700?, 0xc001396480?})
	/workspace/src/github.com/tektoncd/cli/pkg/pipelinerun/tracker.go:111 +0x2d
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/controller.go:239
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/shared_informer.go:816 +0xf7
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x1c00032a640?)
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x46ee67?, {0x38dea60, 0xc001358000}, 0x1, 0xc0009ac000)
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x1b?, 0x3b9aca00, 0x0, 0xa0?, 0xc00032a788?)
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000206d80?)
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/client-go/tools/cache/shared_informer.go:812 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/workspace/src/github.com/tektoncd/cli/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x85

It occurrs mostly at the end of the pipeline I see.

EDIT: It occurs on older version v0.30.0, I bumped also there to v0.31.1 with a hope that #2048 will resolve it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
0.11.0
Development

No branches or pull requests
5 participants
@chmouel @keskad @piyush-garg @pradeepitm12 @tekton-robot