Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvnemesis: TestKVNemesisMultiNode failed #116541

Closed
cockroach-teamcity opened this issue Dec 15, 2023 · 6 comments
Closed

kv/kvnemesis: TestKVNemesisMultiNode failed #116541

cockroach-teamcity opened this issue Dec 15, 2023 · 6 comments
Assignees
@arulajmani
Labels
branch-release-23.2.0-rc (deleted) C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team X-duplicate Closed as a duplicate of another issue.
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Dec 15, 2023 •
edited by cockroach-jira-scripts
Loading

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2.0-rc @ e5f281408027c52fa0f63dd48401a25a61eccfd7:

Fatal error:

panic: lock table num=6
 lock: /Table/100/"4bb36f9fb69266ca"
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
    active: true req: 44, strength: Exclusive, txn: b97e6119-c1d7-44a6-af05-fe1bc721b955
   distinguished req: 44
 lock: /Table/100/"5cd57decca3b9f3e"
  holder: txn: ec9801ad-33de-4a26-8202-3f1cb3e3e445 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
   distinguished req: 43
 lock: /Table/100/"76ade637c4c98a78"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: true req: 24, strength: Intent, txn: none
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
   distinguished req: 24
 lock: /Table/100/"99156dcc7eed00c4"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
 lock: /Table/100/"9a1e0d4fd5024e4b"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
 lock: /Table/100/"9a9545fcd902fa5a"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97

error: queued locking request 43 does not conflict with holder/waiting requests  lock: /Table/100/"5cd57decca3b9f3e"
  holder: txn: ec9801ad-33de-4a26-8202-3f1cb3e3e445 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
   distinguished req: 43
	panic: lock table num=6
 lock: /Table/100/"4bb36f9fb69266ca"
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
    active: true req: 44, strength: Exclusive, txn: b97e6119-c1d7-44a6-af05-fe1bc721b955
   distinguished req: 44
 lock: /Table/100/"5cd57decca3b9f3e"
  holder: txn: ec9801ad-33de-4a26-8202-3f1cb3e3e445 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
   distinguished req: 43
 lock: /Table/100/"76ade637c4c98a78"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
 lock: /Table/100/"99156dcc7eed00c4"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
 lock: /Table/100/"9a1e0d4fd5024e4b"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
 lock: /Table/100/"9a9545fcd902fa5a"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97

error: queued locking request 43 does not conflict with holder/waiting requests  lock: /Table/100/"5cd57decca3b9f3e"
  holder: txn: ec9801ad-33de-4a26-8202-3f1cb3e3e445 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
   distinguished req: 43 [recovered]
	panic: lock table num=6
 lock: /Table/100/"4bb36f9fb69266ca"
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
    active: true req: 44, strength: Exclusive, txn: b97e6119-c1d7-44a6-af05-fe1bc721b955
   distinguished req: 44
 lock: /Table/100/"5cd57decca3b9f3e"
  holder: txn: ec9801ad-33de-4a26-8202-3f1cb3e3e445 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
   distinguished req: 43
 lock: /Table/100/"76ade637c4c98a78"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
 lock: /Table/100/"99156dcc7eed00c4"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
 lock: /Table/100/"9a1e0d4fd5024e4b"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
 lock: /Table/100/"9a9545fcd902fa5a"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97

error: queued locking request 43 does not conflict with holder/waiting requests  lock: /Table/100/"5cd57decca3b9f3e"
  holder: txn: ec9801ad-33de-4a26-8202-3f1cb3e3e445 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
   distinguished req: 43 [recovered]
	panic: lock table num=6
 lock: /Table/100/"4bb36f9fb69266ca"
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
    active: true req: 44, strength: Exclusive, txn: b97e6119-c1d7-44a6-af05-fe1bc721b955
   distinguished req: 44
 lock: /Table/100/"5cd57decca3b9f3e"
  holder: txn: ec9801ad-33de-4a26-8202-3f1cb3e3e445 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
   distinguished req: 43
 lock: /Table/100/"76ade637c4c98a78"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
 lock: /Table/100/"99156dcc7eed00c4"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
 lock: /Table/100/"9a1e0d4fd5024e4b"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
 lock: /Table/100/"9a9545fcd902fa5a"
  holder: txn: 25986d71-fc44-4ea8-8499-42a8faff190f epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: false req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97

error: queued locking request 43 does not conflict with holder/waiting requests  lock: /Table/100/"5cd57decca3b9f3e"
  holder: txn: ec9801ad-33de-4a26-8202-3f1cb3e3e445 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 43, strength: Shared, txn: 26b9059d-df9b-4fe7-a102-86cbd23d0d97
   distinguished req: 43

Stack:

goroutine 43474 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc0047a4000?, {0x6b51e80, 0xc012e2b5c0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x4690d60?, 0xc00ab76680?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1()
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:109 +0x14e
panic({0x4690d60?, 0xc00ab76680?})
	GOROOT/src/runtime/panic.go:920 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc0066d4c80)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x1d4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.Dequeue({{0x7f6593dda938?, 0xc0066d4c80?}}, {0x6b805b8?, 0xc00b074280?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:77 +0x86
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).FinishReq(0xc0099e1360?, 0xc004a80480)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:453 +0x102
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries.func1()
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:434 +0x32
panic({0x4690d60?, 0xc00ab25d80?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc0066d4c80)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x1d4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.ScanAndEnqueue({{0x7f6593dda938?, 0xc0066d4c80?}}, {0x0, {0x17a104d4cbc3252e, 0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:64 +0x131
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc0099e1360, {0x6b51e80, 0xc012e2b950}, 0xc004a80480, 0x1ace42f?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:337 +0x856
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0x2?, {0x6b51e80, 0xc012e2b950}, 0xc004a80480?, {0x0, {0x17a104d540a8971c, 0x2, 0x0}, 0x0, 0x0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:241 +0x2cc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc010e81900, {0x6b51e80, 0xc012e2b950}, 0xc00ac9db00, 0x52cd588)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x32c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc010e81900, {0x6b51e80?, 0xc012e2b7d0?}, 0xc00ac9db00)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:189 +0x725
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc002cd3500, {0x6b51e80?, 0xc012e2b710?}, 0xc00ac9db00)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x85c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0x1?, {0x6b51e80, 0xc012e2b710}, 0xc00ac9db00)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0xef
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc003ee0400, {0x6b51e80?, 0xc012e2b680?}, {0xc003483680?}, 0xc00ac9db00)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1328 +0x545
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc003ee0400, {0x6b51e80, 0xc012e2b5c0}, 0xc00ac9db00)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1462 +0x2b5
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0x6b51e80, 0xc012e2b5c0}, {0x4f1bac0?, 0xc00ac9db00})
	github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10416 +0x72
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x6b51e80, 0xc012e2b5c0}, {0x4f1bac0, 0xc00ac9db00}, 0xc0072e8b80, 0xc006fdd638)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x4b9
google.golang.org/grpc.getChainUnaryHandler.func1({0x6b51e80, 0xc012e2b5c0}, {0x4f1bac0, 0xc00ac9db00})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x6b51e80, 0xc012e2b5c0}, {0x4f1bac0, 0xc00ac9db00}, 0xc0072e8b80?, 0xc009856b00)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x76
google.golang.org/grpc.getChainUnaryHandler.func1({0x6b51e80, 0xc012e2b5c0}, {0x4f1bac0, 0xc00ac9db00})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc0047a4000?, {{0x424bc5?}, {0x6b88bd0?, 0xc008c577a0?}}}, {0x6b51e80, 0xc012e2b5c0}, {0x4f1bac0, 0xc00ac9db00}, 0xc0072e8b80, 0xc009856a80)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105 +0x273
google.golang.org/grpc.getChainUnaryHandler.func1({0x6b51e80, 0xc012e2b5c0}, {0x4f1bac0, 0xc00ac9db00})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x6b51e80?, 0xc012e2b5c0?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x36
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0023bf170, {0x6b51e80?, 0xc012e2b5c0}, {0x0?, 0xc00aef1978?}, 0xc00aef1970)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd3
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x6b51e80?, 0xc012e2b5c0?}, {0x4f1bac0?, 0xc00ac9db00?}, 0xc0072e8b80?, 0xc006fdd638?)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x8f
google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x6b51e80, 0xc012e2b5c0}, {0x4f1bac0, 0xc00ac9db00}, 0xc00fa6da48?, 0x484ae60?)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0x85
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler({0x4ef9480?, 0xc003ee0400}, {0x6b51e80, 0xc012e2b5c0}, 0xc003a66930, 0xc0026f36c0)
	github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10418 +0x135
google.golang.org/grpc.(*Server).processUnaryRPC(0xc005d6d4a0, {0x6b99ae0, 0xc004733520}, 0xc00ac9d9e0, 0xc007f3f920, 0x9864180, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336 +0xd67
google.golang.org/grpc.(*Server).handleStream(0xc005d6d4a0, {0x6b99ae0, 0xc004733520}, 0xc00ac9d9e0, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704 +0x9e7
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 +0x8d
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 38112
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:963 +0x246
Log preceding fatal error 

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode3226918342
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 4903028559405478051
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1532348750

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-34666
@cockroach-teamcity cockroach-teamcity added branch-release-23.2.0-rc (deleted) C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Dec 15, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Dec 15, 2023
@andrewbaptist
Copy link
Collaborator

Consolidating over at #115694. Removing release blocker.

@andrewbaptist andrewbaptist added X-duplicate Closed as a duplicate of another issue. and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Dec 15, 2023
@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2.0-rc @ 0b0851f59e6c37706c5df624fa3a579faa46d736:

Fatal error:

panic: lock table num=4
 lock: /Table/100/"3ecbf56c82ebf8cf"
   queued locking requests:
    active: false req: 49, strength: Shared, txn: 2f3a537b-c021-42c3-9272-3f6f6fd0db0d
 lock: /Table/100/"3ff63d759589a1a2"
  holder: txn: ef672b11-cc60-4e6e-8ec8-1341e1a8545f epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 49, strength: Shared, txn: 2f3a537b-c021-42c3-9272-3f6f6fd0db0d
   distinguished req: 49
 lock: /Table/100/"42f4c4bd390b14b6"
  holder: txn: ef672b11-cc60-4e6e-8ec8-1341e1a8545f epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"4476c4da411e1b3a"
  holder: txn: ef672b11-cc60-4e6e-8ec8-1341e1a8545f epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]

error: queued locking request 49 does not conflict with holder/waiting requests  lock: /Table/100/"3ff63d759589a1a2"
  holder: txn: ef672b11-cc60-4e6e-8ec8-1341e1a8545f epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: true req: 49, strength: Shared, txn: 2f3a537b-c021-42c3-9272-3f6f6fd0db0d
   distinguished req: 49 [recovered]
	panic: lock table num=4
 lock: /Table/100/"3ecbf56c82ebf8cf"
   queued locking requests:
    active: false req: 49, strength: Shared, txn: 2f3a537b-c021-42c3-9272-3f6f6fd0db0d
 lock: /Table/100/"3ff63d759589a1a2"
  holder: txn: ef672b11-cc60-4e6e-8ec8-1341e1a8545f epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 49, strength: Shared, txn: 2f3a537b-c021-42c3-9272-3f6f6fd0db0d
   distinguished req: 49
 lock: /Table/100/"42f4c4bd390b14b6"
  holder: txn: ef672b11-cc60-4e6e-8ec8-1341e1a8545f epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"4476c4da411e1b3a"
  holder: txn: ef672b11-cc60-4e6e-8ec8-1341e1a8545f epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]

error: queued locking request 49 does not conflict with holder/waiting requests  lock: /Table/100/"3ff63d759589a1a2"
  holder: txn: ef672b11-cc60-4e6e-8ec8-1341e1a8545f epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: true req: 49, strength: Shared, txn: 2f3a537b-c021-42c3-9272-3f6f6fd0db0d
   distinguished req: 49

Stack:

goroutine 7207036 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1ee2da5?, {0x6b69860, 0xc009f4f950})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x46a5c40?, 0xc01bea2ab0?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc00f180a00)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x20d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.UpdateLocks({{0x7f0b62541158?, 0xc00f180a00?}}, 0x2?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:99 +0x89
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).OnLockUpdated(0x7f0bb0a7a878?, {0x6b69860, 0xc00e41fec0}, 0xd3eda5?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:552 +0x30
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc00f278c80, {0x6b69860, 0xc00e41fec0}, {0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:793 +0x884
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects(0xc00f278d88, {0x6b69860, 0xc00e41fec0}, {0x6ba8998?, 0xc00ad67908})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:222 +0x828
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter({0x7f0b62544830?, 0xc00f279168}, 0xc007972560)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214 +0x148
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc007972ae8, {0x6b69860, 0xc013bad530}, {0x6b98a30, 0xc00f279108})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295 +0x1e8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc007972ae8, {0x6b69860, 0xc013bad530})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x95
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1053 +0x1445
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:740 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc0020f5500, 0x6b69860?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x12b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc013438e00, {0x6b69860, 0xc009f4f950}, {0x6b4da80, 0xc0020f5500}, 0xc0020f4a80)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x18d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6b69860?, 0xc009f4f950?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 6455825
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415
Log preceding fatal error 

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2464688814
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 3767615224077329071
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2649088346

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2.0-rc @ 0b0851f59e6c37706c5df624fa3a579faa46d736:

Fatal error:

panic: lock table num=4
 lock: /Table/100/"d6f2061c0da643c9"
   queued locking requests:
    active: false req: 18, strength: Shared, txn: 61b5fb05-7c2a-4369-9637-be591c2f7f95
 lock: /Table/100/"dc0d45fd8ed28b7c"
  holder: txn: 8713ea87-08cc-4262-8c7f-8318e46c10cf epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 18, strength: Shared, txn: 61b5fb05-7c2a-4369-9637-be591c2f7f95
   distinguished req: 18
 lock: /Table/100/"e5f4236f5e3c8774"
  holder: txn: 8713ea87-08cc-4262-8c7f-8318e46c10cf epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 17, strength: Intent, txn: f405e7a5-254a-49ce-8a9d-7c398f965e35
   distinguished req: 17
 lock: /Table/100/"e66a723c04b4d220"
  holder: txn: 8713ea87-08cc-4262-8c7f-8318e46c10cf epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 1, strength: Exclusive, txn: 5badb6ed-5071-474f-859c-6cbb7451d0ae
   distinguished req: 1

error: queued locking request 18 does not conflict with holder/waiting requests  lock: /Table/100/"dc0d45fd8ed28b7c"
  holder: txn: 8713ea87-08cc-4262-8c7f-8318e46c10cf epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: true req: 18, strength: Shared, txn: 61b5fb05-7c2a-4369-9637-be591c2f7f95
   distinguished req: 18 [recovered]
	panic: lock table num=4
 lock: /Table/100/"d6f2061c0da643c9"
   queued locking requests:
    active: false req: 18, strength: Shared, txn: 61b5fb05-7c2a-4369-9637-be591c2f7f95
 lock: /Table/100/"dc0d45fd8ed28b7c"
  holder: txn: 8713ea87-08cc-4262-8c7f-8318e46c10cf epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 18, strength: Shared, txn: 61b5fb05-7c2a-4369-9637-be591c2f7f95
   distinguished req: 18
 lock: /Table/100/"e5f4236f5e3c8774"
  holder: txn: 8713ea87-08cc-4262-8c7f-8318e46c10cf epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 17, strength: Intent, txn: f405e7a5-254a-49ce-8a9d-7c398f965e35
   distinguished req: 17
 lock: /Table/100/"e66a723c04b4d220"
  holder: txn: 8713ea87-08cc-4262-8c7f-8318e46c10cf epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 1, strength: Exclusive, txn: 5badb6ed-5071-474f-859c-6cbb7451d0ae
   distinguished req: 1

error: queued locking request 18 does not conflict with holder/waiting requests  lock: /Table/100/"dc0d45fd8ed28b7c"
  holder: txn: 8713ea87-08cc-4262-8c7f-8318e46c10cf epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: true req: 18, strength: Shared, txn: 61b5fb05-7c2a-4369-9637-be591c2f7f95
   distinguished req: 18 [recovered]
	panic: lock table num=4
 lock: /Table/100/"d6f2061c0da643c9"
   queued locking requests:
    active: false req: 18, strength: Shared, txn: 61b5fb05-7c2a-4369-9637-be591c2f7f95
 lock: /Table/100/"dc0d45fd8ed28b7c"
  holder: txn: 8713ea87-08cc-4262-8c7f-8318e46c10cf epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 18, strength: Shared, txn: 61b5fb05-7c2a-4369-9637-be591c2f7f95
   distinguished req: 18
 lock: /Table/100/"e5f4236f5e3c8774"
  holder: txn: 8713ea87-08cc-4262-8c7f-8318e46c10cf epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 17, strength: Intent, txn: f405e7a5-254a-49ce-8a9d-7c398f965e35
   distinguished req: 17
 lock: /Table/100/"e66a723c04b4d220"
  holder: txn: 8713ea87-08cc-4262-8c7f-8318e46c10cf epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 1, strength: Exclusive, txn: 5badb6ed-5071-474f-859c-6cbb7451d0ae
   distinguished req: 1

error: queued locking request 18 does not conflict with holder/waiting requests  lock: /Table/100/"dc0d45fd8ed28b7c"
  holder: txn: 8713ea87-08cc-4262-8c7f-8318e46c10cf epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: true req: 18, strength: Shared, txn: 61b5fb05-7c2a-4369-9637-be591c2f7f95
   distinguished req: 18

Stack:

goroutine 39645 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc0026ea000?, {0x6b62740, 0xc012bf6c00})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x469d360?, 0xc001295e90?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1()
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:109 +0x14e
panic({0x469d360?, 0xc001295e90?})
	GOROOT/src/runtime/panic.go:920 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc00146ea00)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x1d4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.ScanAndEnqueue({{0x7fb9a4e3c270?, 0xc00146ea00?}}, {0x0, {0x17a5e9828ea938d4, 0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:64 +0x131
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc009ff1770, {0x6b62740, 0xc012bf6f00}, 0xc007554300, 0x0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:337 +0x856
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0x2?, {0x6b62740, 0xc012bf6f00}, 0x1?, {0x0, {0x17a5e9828ea938d4, 0x0, 0x0}, 0x0, 0x0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:241 +0x2cc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc01354d200, {0x6b62740, 0xc012bf6f00}, 0xc003abe000, 0x52dc1f8)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x32c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc01354d200, {0x6b62740?, 0xc012bf6de0?}, 0xc003abe000)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:189 +0x725
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc004034000, {0x6b62740?, 0xc012bf6d20?}, 0xc003abe000)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x85c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0x1?, {0x6b62740, 0xc012bf6d20}, 0xc003abe000)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0xef
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc003e2f400, {0x6b62740?, 0xc012bf6cc0?}, {0xc003309f80?}, 0xc003abe000)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1328 +0x545
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc003e2f400, {0x6b62740, 0xc012bf6c00}, 0xc003abe000)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1462 +0x2b5
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0x6b62740, 0xc012bf6c00}, {0x4f29f00?, 0xc003abe000})
	github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10416 +0x72
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x6b62740, 0xc012bf6c00}, {0x4f29f00, 0xc003abe000}, 0xc017038a60, 0xc008e945a0)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x4b9
google.golang.org/grpc.getChainUnaryHandler.func1({0x6b62740, 0xc012bf6c00}, {0x4f29f00, 0xc003abe000})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x6b62740, 0xc012bf6c00}, {0x4f29f00, 0xc003abe000}, 0xc017038a60?, 0xc007cd8f40)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x76
google.golang.org/grpc.getChainUnaryHandler.func1({0x6b62740, 0xc012bf6c00}, {0x4f29f00, 0xc003abe000})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc0026ea000?, {{0x424bc5?}, {0x6b99730?, 0xc00b353f80?}}}, {0x6b62740, 0xc012bf6c00}, {0x4f29f00, 0xc003abe000}, 0xc017038a60, 0xc007cd8cc0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105 +0x273
google.golang.org/grpc.getChainUnaryHandler.func1({0x6b62740, 0xc012bf6c00}, {0x4f29f00, 0xc003abe000})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x6b62740?, 0xc012bf6c00?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x36
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00389f050, {0x6b62740?, 0xc012bf6c00}, {0x0?, 0xc008329978?}, 0xc008329970)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd3
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x6b62740?, 0xc012bf6c00?}, {0x4f29f00?, 0xc003abe000?}, 0xc017038a60?, 0xc008e945a0?)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x8f
google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x6b62740, 0xc012bf6c00}, {0x4f29f00, 0xc003abe000}, 0xc00f0eca48?, 0x4857de0?)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0x85
github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler({0x4f078a0?, 0xc003e2f400}, {0x6b62740, 0xc012bf6c00}, 0xc004ae5110, 0xc005aacfc0)
	github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10418 +0x135
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0067b9e00, {0x6baa620, 0xc00a28f6c0}, 0xc0055b1b00, 0xc007e6faa0, 0x987b240, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336 +0xd67
google.golang.org/grpc.(*Server).handleStream(0xc0067b9e00, {0x6baa620, 0xc00a28f6c0}, 0xc0055b1b00, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704 +0x9e7
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 +0x8d
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 29260
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:963 +0x246
Log preceding fatal error 

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode3782622087
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 4521731759763184779
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis511866121

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2.0-rc @ 8496f8fa0f076f739862b6e9ceace125a8eb2d93:

Fatal error:

panic: lock table num=5
 lock: /Table/100/"5f4fb6567df04796"
  holder: txn: e8358907-20d5-41b7-96d1-99faaa7bd2c0 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"9a741a3a9317ec6f"
  holder: txn: e8358907-20d5-41b7-96d1-99faaa7bd2c0 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"b35b9c65f6e96983"
  holder: txn: e8358907-20d5-41b7-96d1-99faaa7bd2c0 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 124, strength: Shared, txn: 7343db3e-bc51-4d14-bce7-6d05b1b9d61b
   distinguished req: 124
 lock: /Table/100/"bd01c0d6c8fc8768"
  holder: txn: e8358907-20d5-41b7-96d1-99faaa7bd2c0 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"c16a06e0eb00cb70"
  holder: txn: e8358907-20d5-41b7-96d1-99faaa7bd2c0 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]

error: queued locking request 124 does not conflict with holder/waiting requests  lock: /Table/100/"b35b9c65f6e96983"
  holder: txn: e8358907-20d5-41b7-96d1-99faaa7bd2c0 epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: true req: 124, strength: Shared, txn: 7343db3e-bc51-4d14-bce7-6d05b1b9d61b
   distinguished req: 124 [recovered]
	panic: lock table num=5
 lock: /Table/100/"5f4fb6567df04796"
  holder: txn: e8358907-20d5-41b7-96d1-99faaa7bd2c0 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"9a741a3a9317ec6f"
  holder: txn: e8358907-20d5-41b7-96d1-99faaa7bd2c0 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"b35b9c65f6e96983"
  holder: txn: e8358907-20d5-41b7-96d1-99faaa7bd2c0 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 124, strength: Shared, txn: 7343db3e-bc51-4d14-bce7-6d05b1b9d61b
   distinguished req: 124
 lock: /Table/100/"bd01c0d6c8fc8768"
  holder: txn: e8358907-20d5-41b7-96d1-99faaa7bd2c0 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"c16a06e0eb00cb70"
  holder: txn: e8358907-20d5-41b7-96d1-99faaa7bd2c0 epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]

error: queued locking request 124 does not conflict with holder/waiting requests  lock: /Table/100/"b35b9c65f6e96983"
  holder: txn: e8358907-20d5-41b7-96d1-99faaa7bd2c0 epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: true req: 124, strength: Shared, txn: 7343db3e-bc51-4d14-bce7-6d05b1b9d61b
   distinguished req: 124

Stack:

goroutine 6684880 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1ee2da5?, {0x6b69860, 0xc0011d4240})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x46a5c40?, 0xc014d65a80?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc018419400)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x20d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.UpdateLocks({{0x7f363e040db0?, 0xc018419400?}}, 0x2?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:99 +0x89
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).OnLockUpdated(0x7f363e105108?, {0x6b69860, 0xc015e98120}, 0xd3eda5?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:552 +0x30
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc0052bc580, {0x6b69860, 0xc015e98120}, {0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:793 +0x884
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects(0xc0052bc688, {0x6b69860, 0xc015e98120}, {0x6ba8998?, 0xc00e276f08})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:222 +0x828
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter({0x7f35f28619d0?, 0xc0052bca68}, 0xc0169d2560)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214 +0x148
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc0169d2ae8, {0x6b69860, 0xc00e2e3ad0}, {0x6b98a30, 0xc0052bca08})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295 +0x1e8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc0169d2ae8, {0x6b69860, 0xc00e2e3ad0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x95
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1053 +0x1445
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:740 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc004696a80, 0x6b69860?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x12b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc00c1bea10, {0x6b69860, 0xc0011d4240}, {0x6b4da80, 0xc004696a80}, 0xc004a90000)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x18d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6b69860?, 0xc0011d4240?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 6652648
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415
Log preceding fatal error 

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode3843676885
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 6424800741909141085
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis755824096

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 5, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2.0-rc @ 1159a0f9816f3b4768e4988ad83f8d9eff9a3183:

Fatal error:

panic: raft closed timestamp regression in cmd: 450747ef87ce7014 (term: 6, index: 15); batch state: 1704640779.402664522,0, command: 1704640779.372779513,0, lease: repl=(n1,s1):1 seq=9 start=1704640762.972627924,0 epo=1 pro=1704640762.992898758,0, req: <unknown; not leaseholder>, applying at LAI: 2.
Closed timestamp was set by req: <unknown; not leaseholder or not lease request> under lease: repl=(n1,s1):1 seq=9 start=1704640762.972627924,0 epo=1 pro=1704640762.992898758,0; applied at LAI: 1. Batch idx: 3.
Raft log tail:
0,0 /Local/RangeID/80/u/RaftLog/logIndex:15 (0x0169d8757266746c000000000000000f00): Term:6 Index:15 Type:EntryNormal  (ID 450747ef87ce7014) by lease #9
proposer_lease_sequence:9 max_lease_index:2 closed_timestamp:<wall_time:1704640779372779513 > replicated_eval_result:<write_timestamp:<> delta:<last_update_nanos:1704640782069512589 lock_bytes:1965 lock_count:15 > > logical_op_log:<> trace_data:<key:"crdb-tracer-spanid" value:"3d793af5570d54ce" > trace_data:<key:"crdb-tracer-traceid" value:"4aeacf70a5a836fb" > trace_data:<key:"rec" value:"v" > 
write batch:
Put: /Local/Lock/Table/100/"90a6e44592272eae" 028e8247f7aba94709bfb2498ce50138a6 (0x017a6b12ec123930613665343435393232373265616500ff01000100028e8247f7aba94709bfb2498ce50138a612): 1704640782.069512589,0 txn={id=8e8247f7 key=/Table/100/"76e9698edb358e57" iso=Serializable pri=0.02717194 epo=0 ts=1704640782.069512589,0 min=1704640782.069512589,0 seq=0} ts=1704640782.069512589,0 del=false klen=0 vlen=0 mergeTs=<nil> txnDidNotUpdateMeta=true
Put: /Local/Lock/Table/100/"93f953d6ff37f4c5" 028e8247f7aba94709bfb2498ce50138a6 (0x017a6b12ec123933663935336436666633376634633500ff01000100028e8247f7aba94709bfb2498ce50138a612): 1704640782.069512589,0 txn={id=8e8247f7 key=/Table/100/"76e9698edb358e57" iso=Serializable pri=0.02717194 epo=0 ts=1704640782.069512589,0 min=1704640782.069512589,0 seq=0} ts=1704640782.069512589,0 del=false klen=0 vlen=0 mergeTs=<nil> txnDidNotUpdateMeta=true
Put: /Local/Lock/Table/100/"94522f2dca93b69b" 028e8247f7aba94709bfb2498ce50138a6 (0x017a6b12ec123934353232663264636139336236396200ff01000100028e8247f7aba94709bfb2498ce50138a612): 1704640782.069512589,0 txn={id=8e8247f7 key=/Table/100/"76e9698edb358e57" iso=Serializable pri=0.02717194 epo=0 ts=1704640782.069512589,0 min=1704640782.069512589,0 seq=0} ts=1704640782.069512589,0 del=false klen=0 vlen=0 mergeTs=<nil> txnDidNotUpdateMeta=true
Put: /Local/Lock/Table/100/"9b327723f6d867ac" 028e8247f7aba94709bfb2498ce50138a6 (0x017a6b12ec12396233323737323366...
0,0 /Local/RangeID/80/u/RaftLog/logIndex:14 (0x0169d8757266746c000000000000000e00): Term:6 Index:14 Type:EntryNormal  (ID 6f1758389525834e) by lease #9
proposer_lease_sequence:9 max_lease_index:1 closed_timestamp:<wall_time:1704640779402664522 > replicated_eval_result:<write_timestamp:<> delta:<last_update_nanos:1704640782384740324 sys_bytes:-203 sys_count:-1 > > logical_op_log:<> 
write batch:
Delete (Sized at 203): 0,0 /Local/Range/Table/100/"771b8cf25ebb8ab6"/Transaction/"43c752f6-914e-41f5-b85f-276a7e0e5d84" (0x016b12ec123737316238636632356562623861623600ff01000174786e2d43c752f6914e41f5b85f276a7e0e5d8400): 
0,0 /Local/RangeID/80/u/RaftLog/logIndex:13 (0x0169d8757266746c000000000000000d00): Term:6 Index:13 Type:EntryNormal  (ID 5c2068c6aee7e0e0) by lease #9
proposer_lease_sequence:9 max_lease_index:1 closed_timestamp:<wall_time:1704640779361734754 > replicated_eval_result:<write_timestamp:<wall_time:1704640782278486320 > delta:<last_update_nanos:1704640782278486320 live_bytes:53 live_count:1 key_bytes:33 key_count:1 val_bytes:20 val_count:1 > > logical_op_log:<ops:<write_value:<key:"\354\022765ca81c7323a8fd\000\001" timestamp:<wall_time:1704640782278486320 > > > > trace_data:<key:"crdb-tracer-spanid" value:"25937f17ecefde74" > trace_data:<key:"crdb-tracer-traceid" value:"3c6e127eb0ccf" > trace_data:<key:"rec" value:"v" > 
write batch:
Put: 1704640782.278486320,0 /Table/100/"765ca81c7323a8fd" (0xec123736356361383163373332336138666400010017a819c2895a693009): "\x00\x00\x00\x06e\n\x00\x12\x02\x8e\x01>\xcb\xc1\xa2\x03v142"
0,0 /Local/RangeID/80/u/RaftLog/logIndex:12 (0x0169d8757266746c000000000000000c00): Term:6 Index:12 Type:EntryNormal  (ID 6f1758389525834e) by lease #9
proposer_lease_sequence:9 max_lease_index:1 closed_timestamp:<wall_time:1704640779402664522 > replicated_eval_result:<write_timestamp:<> delta:<last_update_nanos:1704640782384740324 sys_bytes:-203 sys_count:-1 > > logical_op_log:<> 
write batch:
Delete (Sized at 203): 0,0 /Local/Range/Table/100/"771b8cf25ebb8ab6"/Transaction/"43c752f6-914e-41f5-b85f-276a7e0e5d84" (0x016b12ec123737316238636632356562623861623600ff01000174786e2d43c752f6914e41f5b85f276a7e0e5d8400): 
0,0 /Local/RangeID/80/u/RaftLog/logIndex:11 (0x0169d8757266746c000000000000000b00): Term:6 Index:11 Type:EntryNormal : EMPTY [recovered]
	panic: raft closed timestamp regression in cmd: 450747ef87ce7014 (term: 6, index: 15); batch state: 1704640779.402664522,0, command: 1704640779.372779513,0, lease: repl=(n1,s1):1 seq=9 start=1704640762.972627924,0 epo=1 pro=1704640762.992898758,0, req: <unknown; not leaseholder>, applying at LAI: 2.
Closed timestamp was set by req: <unknown; not leaseholder or not lease request> under lease: repl=(n1,s1):1 seq=9 start=1704640762.972627924,0 epo=1 pro=1704640762.992898758,0; applied at LAI: 1. Batch idx: 3.
Raft log tail:
0,0 /Local/RangeID/80/u/RaftLog/logIndex:15 (0x0169d8757266746c000000000000000f00): Term:6 Index:15 Type:EntryNormal  (ID 450747ef87ce7014) by lease #9
proposer_lease_sequence:9 max_lease_index:2 closed_timestamp:<wall_time:1704640779372779513 > replicated_eval_result:<write_timestamp:<> delta:<last_update_nanos:1704640782069512589 lock_bytes:1965 lock_count:15 > > logical_op_log:<> trace_data:<key:"crdb-tracer-spanid" value:"3d793af5570d54ce" > trace_data:<key:"crdb-tracer-traceid" value:"4aeacf70a5a836fb" > trace_data:<key:"rec" value:"v" > 
write batch:
Put: /Local/Lock/Table/100/"90a6e44592272eae" 028e8247f7aba94709bfb2498ce50138a6 (0x017a6b12ec123930613665343435393232373265616500ff01000100028e8247f7aba94709bfb2498ce50138a612): 1704640782.069512589,0 txn={id=8e8247f7 key=/Table/100/"76e9698edb358e57" iso=Serializable pri=0.02717194 epo=0 ts=1704640782.069512589,0 min=1704640782.069512589,0 seq=0} ts=1704640782.069512589,0 del=false klen=0 vlen=0 mergeTs=<nil> txnDidNotUpdateMeta=true
Put: /Local/Lock/Table/100/"93f953d6ff37f4c5" 028e8247f7aba94709bfb2498ce50138a6 (0x017a6b12ec123933663935336436666633376634633500ff01000100028e8247f7aba94709bfb2498ce50138a612): 1704640782.069512589,0 txn={id=8e8247f7 key=/Table/100/"76e9698edb358e57" iso=Serializable pri=0.02717194 epo=0 ts=1704640782.069512589,0 min=1704640782.069512589,0 seq=0} ts=1704640782.069512589,0 del=false klen=0 vlen=0 mergeTs=<nil> txnDidNotUpdateMeta=true
Put: /Local/Lock/Table/100/"94522f2dca93b69b" 028e8247f7aba94709bfb2498ce50138a6 (0x017a6b12ec123934353232663264636139336236396200ff01000100028e8247f7aba94709bfb2498ce50138a612): 1704640782.069512589,0 txn={id=8e8247f7 key=/Table/100/"76e9698edb358e57" iso=Serializable pri=0.02717194 epo=0 ts=1704640782.069512589,0 min=1704640782.069512589,0 seq=0} ts=1704640782.069512589,0 del=false klen=0 vlen=0 mergeTs=<nil> txnDidNotUpdateMeta=true
Put: /Local/Lock/Table/100/"9b327723f6d867ac" 028e8247f7aba94709bfb2498ce50138a6 (0x017a6b12ec12396233323737323366...
0,0 /Local/RangeID/80/u/RaftLog/logIndex:14 (0x0169d8757266746c000000000000000e00): Term:6 Index:14 Type:EntryNormal  (ID 6f1758389525834e) by lease #9
proposer_lease_sequence:9 max_lease_index:1 closed_timestamp:<wall_time:1704640779402664522 > replicated_eval_result:<write_timestamp:<> delta:<last_update_nanos:1704640782384740324 sys_bytes:-203 sys_count:-1 > > logical_op_log:<> 
write batch:
Delete (Sized at 203): 0,0 /Local/Range/Table/100/"771b8cf25ebb8ab6"/Transaction/"43c752f6-914e-41f5-b85f-276a7e0e5d84" (0x016b12ec123737316238636632356562623861623600ff01000174786e2d43c752f6914e41f5b85f276a7e0e5d8400): 
0,0 /Local/RangeID/80/u/RaftLog/logIndex:13 (0x0169d8757266746c000000000000000d00): Term:6 Index:13 Type:EntryNormal  (ID 5c2068c6aee7e0e0) by lease #9
proposer_lease_sequence:9 max_lease_index:1 closed_timestamp:<wall_time:1704640779361734754 > replicated_eval_result:<write_timestamp:<wall_time:1704640782278486320 > delta:<last_update_nanos:1704640782278486320 live_bytes:53 live_count:1 key_bytes:33 key_count:1 val_bytes:20 val_count:1 > > logical_op_log:<ops:<write_value:<key:"\354\022765ca81c7323a8fd\000\001" timestamp:<wall_time:1704640782278486320 > > > > trace_data:<key:"crdb-tracer-spanid" value:"25937f17ecefde74" > trace_data:<key:"crdb-tracer-traceid" value:"3c6e127eb0ccf" > trace_data:<key:"rec" value:"v" > 
write batch:
Put: 1704640782.278486320,0 /Table/100/"765ca81c7323a8fd" (0xec123736356361383163373332336138666400010017a819c2895a693009): "\x00\x00\x00\x06e\n\x00\x12\x02\x8e\x01>\xcb\xc1\xa2\x03v142"
0,0 /Local/RangeID/80/u/RaftLog/logIndex:12 (0x0169d8757266746c000000000000000c00): Term:6 Index:12 Type:EntryNormal  (ID 6f1758389525834e) by lease #9
proposer_lease_sequence:9 max_lease_index:1 closed_timestamp:<wall_time:1704640779402664522 > replicated_eval_result:<write_timestamp:<> delta:<last_update_nanos:1704640782384740324 sys_bytes:-203 sys_count:-1 > > logical_op_log:<> 
write batch:
Delete (Sized at 203): 0,0 /Local/Range/Table/100/"771b8cf25ebb8ab6"/Transaction/"43c752f6-914e-41f5-b85f-276a7e0e5d84" (0x016b12ec123737316238636632356562623861623600ff01000174786e2d43c752f6914e41f5b85f276a7e0e5d8400): 
0,0 /Local/RangeID/80/u/RaftLog/logIndex:11 (0x0169d8757266746c000000000000000b00): Term:6 Index:11 Type:EntryNormal : EMPTY

Stack:

goroutine 6486095 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1ee3085?, {0x6b69b20, 0xc0013b33e0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x4bdc540?, 0xc01788c570?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportOrPanic({0x6b69b20, 0xc0220090b0}, 0xc00af8e000, {0x4f941f9?, 0x100000000?}, {0xc008432208?, 0xc000c88f50?, 0x10?})
	github.com/cockroachdb/cockroach/pkg/util/log/logcrash/crash_reporting.go:417 +0x1e7
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).assertNoCmdClosedTimestampRegression(0xc002558690, {0x6b69b20, 0xc0220090b0}, 0xc0110ea6b8)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:736 +0x546
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage(0xc002558690, {0x6b69b20, 0xc0220090b0}, {0x6b89ac0?, 0xc0110ea6b8})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:117 +0x135
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter({0x6b98cf0?, 0xc002558a38}, 0xc008432578)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:192 +0x148
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc008432ae8, {0x6b69b20, 0xc01ef0fc20}, {0x6b98cf0, 0xc002558a08})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:284 +0x150
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc008432ae8, {0x6b69b20, 0xc01ef0fc20})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x95
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1053 +0x1445
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:740 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc003696000, 0x6b69b20?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x12b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc00f7be230, {0x6b69b20, 0xc0013b33e0}, {0x6b4dd40, 0xc003696000}, 0xc00e3b7500)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x18d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6b69b20?, 0xc0013b33e0?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 5864881
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415
Log preceding fatal error 

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2234031977
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 5096222463049590446
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2185459920

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@pav-kv pav-kv mentioned this issue Jan 8, 2024
Open
@nvanbenschoten
Copy link
Member

Duplicate of #115694. Closing, as this assertion has been disarmed on the release-23.2.0-rc branch.

@nvanbenschoten nvanbenschoten closed this as not planned Won't fix, can't repro, duplicate, stale Jan 10, 2024
@github-project-automation github-project-automation bot moved this to Closed in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arulajmani arulajmani

Labels
branch-release-23.2.0-rc (deleted) C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team X-duplicate Closed as a duplicate of another issue.
Projects
No open projects
KV
Status: Closed
Milestone
23.2
Development

No branches or pull requests
4 participants
@nvanbenschoten @arulajmani @cockroach-teamcity @andrewbaptist