Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cli: TestCollectInfoFromMultipleStores failed #96413

Closed
cockroach-teamcity opened this issue Feb 2, 2023 · 1 comment
Closed

cli: TestCollectInfoFromMultipleStores failed #96413

cockroach-teamcity opened this issue Feb 2, 2023 · 1 comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Feb 2, 2023 •
edited by cockroach-jira-scripts
Loading

cli.TestCollectInfoFromMultipleStores failed with artifacts on master @ 22244a780dcfaca48162dde8e0f90b5ba9b6bb9c:

github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1({0x7cb33e0, 0xc007274000}, {0xb7a64a8, 0xc002febb00})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:898 +0x1ea
github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run({0xc004a102e0, 0x4, 0x4}, {0x7cb33e0, 0xc007274000}, {0xb7a64a8, 0xc002febb00}, 0xe52e0c0, 0xc005612960)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:900 +0x26c
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func2()
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1062 +0x205
created by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1050 +0x4a6

goroutine 526687 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001cd5990, 0x466)
	GOROOT/src/runtime/sema.go:517 +0x14c
sync.(*Cond).Wait(0xc001cd5980)
	GOROOT/src/sync/cond.go:70 +0xa5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006156800, {0xb778398, 0xc0027d1410})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:293 +0x445
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x1f7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x61a

goroutine 526648 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0xb778398, 0xc003752cc0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:842 +0x1bd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x1f7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x61a

goroutine 524449 [chan receive]:
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0xb778398, 0xc0042b8540})
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:324 +0xea
runtime/pprof.Do({0xb778328, 0xc0000820a8}, {{0xc000347300?, 0x1d26880?, 0xc001cfa800?}}, 0xc0019a4fa0)
	GOROOT/src/runtime/pprof/runtime.go:40 +0x123
github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc00288d810)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:322 +0x92
created by github.com/cockroachdb/pebble.(*tableCacheShard).init
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:314 +0x197

goroutine 526665 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001cd5990, 0x492)
	GOROOT/src/runtime/sema.go:517 +0x14c
sync.(*Cond).Wait(0xc001cd5980)
	GOROOT/src/sync/cond.go:70 +0xa5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006156800, {0xb778398, 0xc0027d0bd0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:293 +0x445
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x1f7
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x61a

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-24110
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Feb 2, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Feb 2, 2023
@blathers-crl blathers-crl bot added the T-kv KV Team label Feb 2, 2023
@erikgrinaker
Copy link
Contributor

See cockroachdb/pebble#2300.

=== RUN   TestCollectInfoFromMultipleStores
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1855398a359288315eef18ccd1f2c1dd/logTestCollectInfoFromMultipleStores3531733834
    test_log_scope.go:79: use -show-logs to present logs inline
fatal error: sync: unlock of unlocked mutex

goroutine 533630 [running]:
sync.fatal({0x7dd1bb1?, 0xffffffff?})
	GOROOT/src/runtime/panic.go:1031 +0x1e
sync.(*Mutex).unlockSlow(0xc001ec9078, 0xffffffff)
	GOROOT/src/sync/mutex.go:229 +0x49
sync.(*Mutex).Unlock(0xc001ec9078)
	GOROOT/src/sync/mutex.go:223 +0x55
panic({0x73f4d00, 0xb72f830})
	GOROOT/src/runtime/panic.go:890 +0x262
github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).timeDiskOp(0xc0039012c0, 0x2, 0xc00a7f17a0)
	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:262 +0x1a5
github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).Sync(0xc0039012c0)
	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:219 +0x69
github.com/cockroachdb/pebble/vfs.(*enospcFile).Sync(0xc001f944b0)
	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_full.go:391 +0x6d
github.com/cockroachdb/pebble.(*DB).makeRoomForWrite(0xc001ec8f00, 0x0)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:1978 +0x1559
github.com/cockroachdb/pebble.(*DB).ingest.func1()
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/ingest.go:748 +0x2d2
github.com/cockroachdb/pebble.(*commitPipeline).AllocateSeqNum(0xc001ebf900, 0x6, 0xc00a7f20b0, 0xc00a7f20e8)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/commit.go:385 +0x447
github.com/cockroachdb/pebble.(*DB).ingest(0xc001ec8f00, {0xc005a7ec80, 0x6, 0x8}, 0x80afff8)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/ingest.go:785 +0x5ba
github.com/cockroachdb/pebble.(*DB).IngestWithStats(0xc001ec8f00, {0xc005a7ec80, 0x6, 0x8})
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/ingest.go:680 +0x111
github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).IngestExternalFilesWithStats(0xc00aaad1e0, {0x0?, 0x0?}, {0xc005a7ec80, 0x6, 0x8})
	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1764 +0x66
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).applySnapshot(0xc00844e000, {0xb778398, _}, {{0xbc, 0xa6, 0xfa, 0xb1, 0x32, 0xda, 0x4e, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go:657 +0x1274
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0xbc, 0xa6, 0xfa, 0xb1, 0x32, 0xda, 0x4e, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:806 +0xa0e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1({0xb778398, 0xc00ae13bc0}, 0xc00844e000)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:463 +0x3b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest(0x0?, {0xb778398, 0xc00ae13bc0}, 0xc0054409c8, 0xc00a7f40b8)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:341 +0x15c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest(0xc00224d500, {0xb778398, 0xc00ae13bc0}, 0xc005440960, {{0xbc, 0xa6, 0xfa, 0xb1, 0x32, 0xda, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:404 +0xe5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc00224d500, {0xb778398, 0xc002c9ba40}, 0xc005440960, {0x7fe8c30a13e0, 0xc007ddd9e0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1089 +0xc6e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0xb778398, 0xc0038e3200})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:210 +0xfd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0017d54d0, {0xb778398, 0xc0038e3200}, {0xc0027d0420?, 0x3?}, 0xc00a3a0b68)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:322 +0x148
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc00224d500, {0xb7782f0, 0xc008c7ee80}, 0xc005440960, {0x7fe8c30a13b8, 0xc007ddd9e0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:207 +0x114
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc003dc4500, {0xb7b5930, 0xc007ddd9e0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:375 +0x25b
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x7c54340?, 0xc003dc4500}, {0xb7a9dc0?, 0xc0036f1b20})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:270 +0xc3
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.StreamServerInterceptor.func1({0x7c54340, 0xc003dc4500}, {0xb7aa3a8?, 0xc0023f8380?}, 0xc00a028570, 0x80a9638)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:163 +0x67a
google.golang.org/grpc.chainStreamInterceptors.func1.1({0x7c54340, 0xc003dc4500}, {0xb7aa3a8, 0xc0023f8380})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1482 +0x106
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x7c54340, 0xc003dc4500}, {0xb7aa3a8, 0xc0023f8380}, 0xc00a028570, 0xc008c7edc0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:272 +0xe9
google.golang.org/grpc.chainStreamInterceptors.func1.1({0x7c54340, 0xc003dc4500}, {0xb7aa3a8, 0xc0023f8380})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1485 +0x1ea
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x70508d0?}}}, {0x7c54340, 0xc003dc4500}, {0xb7aa3a8, 0xc0023f8380}, 0xc00a028570, 0xc008c7edc0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:136 +0x468
google.golang.org/grpc.chainStreamInterceptors.func1.1({0x7c54340, 0xc003dc4500}, {0xb7aa3a8, 0xc0023f8380})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1485 +0x1ea
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0xc0017d54d0?, 0x0?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:241 +0x70
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0017d54d0, {0xb778398, 0xc0038e3050}, {0x203002?, 0x203002?}, 0xc0048096c0)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:322 +0x148
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x7c54340, 0xc003dc4500}, {0xb7aa3a8?, 0xc0023f8380?}, 0xc00a028570, 0xc008c7edc0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:240 +0x14d
google.golang.org/grpc.chainStreamInterceptors.func1.1({0x7c54340, 0xc003dc4500}, {0xb7aa3a8, 0xc0023f8380})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1485 +0x1ea
google.golang.org/grpc.chainStreamInterceptors.func1({0x7c54340, 0xc003dc4500}, {0xb7aa3a8, 0xc0023f8380}, 0xc00a028570, 0x80a9638)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1487 +0x276
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00579e000, {0xb7c0880, 0xc00a7c5860}, 0xc0033a9b00, 0xc000a8b500, 0xe561f20, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1ef6
google.golang.org/grpc.(*Server).handleStream(0xc00579e000, {0xb7c0880, 0xc00a7c5860}, 0xc0033a9b00, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1717 +0xfaf
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 +0xed
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:963 +0x4de

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team
Projects
None yet
Milestone
23.1
Development

No branches or pull requests
2 participants
@erikgrinaker @cockroach-teamcity