Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/serverccl: TestServerStartStop failed #100989

Closed
cockroach-teamcity opened this issue Apr 8, 2023 · 3 comments · Fixed by #101296
Closed

ccl/serverccl: TestServerStartStop failed #100989

cockroach-teamcity opened this issue Apr 8, 2023 · 3 comments · Fixed by #101296
Labels
branch-master Failures and bugs on the master branch. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-multitenant Issues owned by the multi-tenant virtual team
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 8, 2023 •
edited by exalate-issue-sync bot
Loading

ccl/serverccl.TestServerStartStop failed with artifacts on master @ 85e41ca8d3d9edacf5ee3061a2591b159b9b0502:

Fatal error:

panic: test timed out after 59m55s

Stack:

goroutine 197815 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2036 +0x8e
created by time.goFunc
	GOROOT/src/time/sleep.go:176 +0x32
Log preceding fatal error 

* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc0089fa770?)
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:306 +0x58
* created by github.com/cockroachdb/pebble.(*tableCacheShard).init
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:298 +0xef
* 
* goroutine 125466 [chan receive, 2 minutes]:
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x72d5948, 0xc0037c1590})
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:308 +0x9f
* runtime/pprof.Do({0x72d58d8?, 0xc000080050?}, {{0xc0002ec9c0?, 0x12056c0?, 0xc012667500?}}, 0xc0089f7fa8)
* 	GOROOT/src/runtime/pprof/runtime.go:40 +0xa3
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc0089f7f70?)
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:306 +0x58
* created by github.com/cockroachdb/pebble.(*tableCacheShard).init
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:298 +0xef
* 
* goroutine 128489 [select]:
* github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0xc0097d5df0)
* 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:128 +0x13e
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc002dae000, {0x72d58a0, 0xc00568dac0}, 0xc013913e50?, {{0xc009b273d8, 0x3, 0x8}, {0xc009b273e0, 0x3, 0x8}}, ...)
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:450 +0x345
* github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.func1.1({0x72d58a0?, 0xc00568dac0?})
* 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:215 +0x105
* github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
* golang.org/x/sync/errgroup.(*Group).Go.func1()
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
* created by golang.org/x/sync/errgroup.(*Group).Go
* 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
* 
* goroutine 125468 [chan receive, 2 minutes]:
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x72d5948, 0xc0037c1650})
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:308 +0x9f
* runtime/pprof.Do({0x72d58d8?, 0xc000080050?}, {{0xc0002ec9c0?, 0x12056c0?, 0xc018130a00?}}, 0xc0052acfa8)
* 	GOROOT/src/runtime/pprof/runtime.go:40 +0xa3
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc0052acf70?)
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:306 +0x58
* created by github.com/cockroachdb/pebble.(*tableCacheShard).init
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:298 +0xef
* 
* goroutine 125471 [chan receive, 2 minutes]:
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x72d5948, 0xc0037c1710})
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:308 +0x9f
* runtime/pprof.Do({0x72d58d8?, 0xc000080050?}, {{0xc0002ec9c0?, 0xc00229f701?, 0x100000001?}}, 0xc0089fafa8)
* 	GOROOT/src/runtime/pprof/runtime.go:40 +0xa3
* github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:306 +0x58
* created by github.com/cockroachdb/pebble.(*tableCacheShard).init
* 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:298 +0xef
* 
*

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-26731

Epic CRDB-27931
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 8, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Apr 8, 2023
@blathers-crl blathers-crl bot added the T-server-and-security DB Server & Security label Apr 8, 2023
@knz knz added T-multitenant Issues owned by the multi-tenant virtual team and removed T-server-and-security DB Server & Security labels Apr 8, 2023
@knz
Copy link
Contributor

knz commented Apr 8, 2023

this repros readily under stress.

@knz knz added GA-blocker branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 labels Apr 8, 2023
@knz
Copy link
Contributor

knz commented Apr 8, 2023

Interestingly -- it only repros if --show-logs is not passed -- i.e. the failure is more likely when the server logs are not displayed in the test logs.

This was referenced Apr 10, 2023
Merged
Merged
Open
@knz knz removed the GA-blocker label Apr 12, 2023
@knz
Copy link
Contributor

knz commented Apr 12, 2023

Removing "GA-blocker" label since the issue is so unlikely to happen on a running server and is likely to only affect unit tests.

@craig craig bot closed this as completed in 773f7de Apr 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-multitenant Issues owned by the multi-tenant virtual team
Projects
None yet
Milestone
23.1
2 participants
@knz @cockroach-teamcity