Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvnemesis: TestKVNemesisSingleNode failed #117854

Closed
cockroach-teamcity opened this issue Jan 17, 2024 · 3 comments · Fixed by #117864
Closed

kv/kvnemesis: TestKVNemesisSingleNode failed #117854

cockroach-teamcity opened this issue Jan 17, 2024 · 3 comments · Fixed by #117864
Assignees
@arulajmani
Labels
A-kv-transactions Relating to MVCC and the transactional model. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jan 17, 2024 •
edited by cockroach-jira-scripts
Loading

kv/kvnemesis.TestKVNemesisSingleNode failed on master @ 1898adc059e9b3026fd5a7ed88dc41cbb006abfc:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1b19252]

Stack:

goroutine 5460 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc0058b5918?, {0x719b570, 0xc0026c31a0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x4a51220?, 0xa2f6b00?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc002830000?, {0x719b570, 0xc0027f57d0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x4a51220?, 0xa2f6b00?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1()
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:109 +0x148
panic({0x4a51220?, 0xa2f6b00?})
	GOROOT/src/runtime/panic.go:920 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableGuardImpl).isSameTxn(...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:804
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableGuardImpl).IsKeyLockedByConflictingTxn(0xc0014f3040, {0xc005ff1d10, 0x14, 0x0?}, 0x1)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:749 +0x452
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*Guard).IsKeyLockedByConflictingTxn(0xc0058b6010?, {0xc005ff1d10?, 0xc0032a3680?, 0xc003d24780?}, 0x0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:809 +0x27
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.(*requestBoundLockTableView).IsKeyLockedByConflictingTxn(0xc005ff1d10?, {0xc005ff1d10?, 0x1a3587b?, 0xc0058b60b0?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/lock.go:255 +0x23
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).isKeyLockedByConflictingTxn(0xc0025ba000, {0x719b570, 0xc0027f5ad0}, {0xc005ff1d10?, 0x7feceda4018c?, 0x8?})
	github.com/cockroachdb/cockroach/pkg/storage/pebble_mvcc_scanner.go:1807 +0xa2
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).add(0xc0025ba000, {0x719b570, 0xc0027f5ad0}, {0xc005ff1d10, 0x14, 0x30}, {0xc005ff1d10, 0x22, 0x30}, {0x7feceda4018c, ...})
	github.com/cockroachdb/cockroach/pkg/storage/pebble_mvcc_scanner.go:1236 +0xc5
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).getOne(0xc0025ba000, {0x719b570, 0xc0027f5ad0})
	github.com/cockroachdb/cockroach/pkg/storage/pebble_mvcc_scanner.go:827 +0xca5
github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).scan(0xc0025ba000, {0x719b570, 0xc0027f5ad0})
	github.com/cockroachdb/cockroach/pkg/storage/pebble_mvcc_scanner.go:658 +0xbc
github.com/cockroachdb/cockroach/pkg/storage.mvccScanToBytes({_, _}, {_, _}, {_, _, _}, {0xc002b6edf8, 0x14, 0x18}, ...)
	github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:4435 +0x350
github.com/cockroachdb/cockroach/pkg/storage.mvccScanToKvs({_, _}, {_, _}, {_, _, _}, {0xc002b6edf8, 0x14, 0x18}, ...)
	github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:4488 +0xfe
github.com/cockroachdb/cockroach/pkg/storage.MVCCScan({_, _}, {_, _}, {_, _, _}, {0xc002b6edf8, 0x14, 0x18}, ...)
	github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:4733 +0x316
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.Scan({_, _}, {_, _}, {{0x7226ec0, 0xc004969940}, {{0x17ab1075b2bc81cd, 0x0}, 0x0, {0x17ab1075b2bcf08a, ...}, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_scan.go:92 +0x95f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand({_, _}, {_, _}, {_, _}, _, _, {{0x17ab1075b2bc81cd, 0x0}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:488 +0x2ea
github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch({_, _}, {_, _}, {_, _}, {_, _}, _, 0xc00481e900, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:305 +0xba5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes(_, {_, _}, {_, _}, {_, _}, _, _, 0xc0058ba4e8, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go:473 +0x3db
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch(0xc000185900, {0x719b570, 0xc0027f5ad0}, 0xc00481e900, 0xc004ca4a90)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go:135 +0x785
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc000185900, {0x719b570, 0xc0027f5ad0}, 0xc00481e900, 0x56093d0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:512 +0x3a3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc000185900, {0x719b570?, 0xc0027f5950?}, 0xc00481e900)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185 +0x6c5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc003ba9500, {0x719b570?, 0xc0027f5890?}, 0xc00481e900)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x85c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0x1?, {0x719b570, 0xc0027f5890}, 0xc00481e900)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0x85
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc00504dc00, {0x719b570?, 0xc0027f5830?}, {0xc0022410e0?}, 0xc00481e900)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1374 +0x545
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc00504dc00, {0x719b570, 0xc0027f57d0}, 0xc00481e900)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1508 +0x2b5
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0x719b570?, 0xc0027f57d0?}, {0x5172720?, 0xc00481e900?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 +0x45
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x719b570, 0xc0027f57d0}, {0x5172720, 0xc00481e900}, 0xc00547fd80, 0xc003925c38)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x4b9
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x719b570?, 0xc0027f57d0?}, {0x5172720?, 0xc00481e900?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x37
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x719b570, 0xc0027f57d0}, {0x5172720, 0xc00481e900}, 0xc0027f57d0?, 0xc00547fda0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x76
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x719b570?, 0xc0027f57d0?}, {0x5172720?, 0xc00481e900?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x37
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc002830000?, {{0x10000004b5d58?}, {0x71d3550?, 0xc005ecac00?}}}, {0x719b570, 0xc0027f57d0}, {0x5172720, 0xc00481e900}, 0xc00547fd80, 0xc00547fdc0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95 +0x1b3
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x719b570?, 0xc0027f57d0?}, {0x5172720?, 0xc00481e900?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x37
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x719b570?, 0xc0027f57d0?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x36
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00436b170, {0x719b570?, 0xc0027f57d0}, {0x0?, 0x1?}, 0xc0022aaa80)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd3
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x719b570?, 0xc0027f57d0?}, {0x5172720?, 0xc00481e900?}, 0x57c325?, 0x30?)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x8f
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x719b570?, 0xc0027f57d0?}, {0x5172720?, 0xc00481e900?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x37
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2({0x719b570?, 0xc0027f57d0?}, {0x0?, 0xc0022aac08?}, {0x5172720?, 0xc00481e900?}, {0x5047d40?, 0xc002921400}, 0xc004b258c0?, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714 +0x51
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8({0x719b570, 0xc0027f57d0}, {0x5303acc, 0x21}, {0x5172720, 0xc00481e900}, {0x5047d40, 0xc002921400}, 0x147ae92?, 0xc0038dcdc0, ...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228 +0x47a
github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1({0x719b570, 0xc0027f57d0}, {0x5303acc, 0x21}, {0x5172720, 0xc00481e900}, {0x5047d40, 0xc002921400}, 0x24?, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899 +0x14e
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3({0x719b570, 0xc0026c31a0}, 0xc00481e7e0, {0x0, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784 +0x32f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc0021ce2a0, {0x719b570, 0xc0026c31a0}, 0xb7?, {0x717fe60, 0xc0044e5920?}, 0xc00481e7e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:209 +0x1bc
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc0021ce2a0, {0x719b570, 0xc0026c31a0}, 0x719b570?)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:187 +0x87
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc0038cd700, {0x719b570, 0xc0026c31a0}, 0xc00479bb00?, {0xc0050ce280, 0xc0015e76c0, 0xc0015e7730, 0x0, 0x0}, 0x0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2407 +0x1136
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc0038cd700, {0x719b570?, 0xc0026c31a0}, 0xc00479bb00, {{0xc003cc94b8, 0x14, 0x18}, {0xc002b6edf8, 0x14, 0x18}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1908 +0x7b2
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatchAsync.func1({0x719b570, 0xc0026c31a0})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1785 +0x147
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 5246
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415
Log preceding fatal error 

=== RUN   TestKVNemesisSingleNode
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestKVNemesisSingleNode1929350409
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:327: seed: 629054170974032836
    kvnemesis_test.go:234: kvnemesis logging to /var/lib/engflow/worker/work/3/exec/_tmp/8b60891bd2789237c716ace82dc4384b/kvnemesis1343475667

Parameters:

  • attempt=1
  • run=10
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-35391
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Jan 17, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Jan 17, 2024
@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisSingleNode failed with artifacts on master @ 1898adc059e9b3026fd5a7ed88dc41cbb006abfc:

        	            	  | github.com/cockroachdb/cockroach/pkg/storage.mvccScanToKvs
        	            	  | 	github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:4488
        	            	  | github.com/cockroachdb/cockroach/pkg/storage.MVCCScan
        	            	  | 	github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:4733
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.ReverseScan
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_reverse_scan.go:89
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:488
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:305
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go:473
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_read.go:135
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:512
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193
        	            	  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1374
        	            	  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1508
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704
        	            	  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
        	            	Wraps: (2) secondary error attachment
        	            	  | runtime error: invalid memory address or nil pointer dereference
        	            	  | (1) runtime error: invalid memory address or nil pointer dereference
        	            	  | Error types: (1) runtime.errorString
        	            	Wraps: (3) panic applying step 
        	            	  | db0.Txn(ctx, func(ctx context.Context, txn *kv.Txn) error {
        	            	  |   txn.SetIsoLevel(isolation.Snapshot)
        	            	  |   txn.GetForUpdate(ctx, tk(2954581848373496430)) // @1705475860.684367306,0 (<nil>, <nil>)
        	            	  |   txn.GetForShare(ctx, tk(574545114681662749)) // @1705475860.684367306,0 (v9, <nil>)
        	            	  |   txn.ReverseScanForShareSkipLocked(ctx, tk(1820498217220570453), tk(7831833396965981956), 0)
        	            	  |   txn.ScanForUpdate(ctx, tk(10391090840890742608), tk(14359564960357966375), 0)
        	            	  |   txn.ReverseScanForUpdate(ctx, tk(7224580107135627516), tk(13038096599029329166), 0)
        	            	  |   txn.GetForShareGuaranteedDurability(ctx, tk(12530610722970564363))
        	            	  |   txn.ScanForShare(ctx, tk(7262151861797917607), tk(18430215113825785752), 0)
        	            	  |   return errors.New("rollback")
        	            	  | }): runtime error: invalid memory address or nil pointer dereference
        	            	Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode3703101735
--- FAIL: TestKVNemesisSingleNode (3.34s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@erikgrinaker
Copy link
Contributor

Same as #117855 (comment).

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisSingleNode failed with artifacts on master @ 1898adc059e9b3026fd5a7ed88dc41cbb006abfc:

F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !goroutine 1066626 [select]:
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:502 +0x14d
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 1054399
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:473 +0x4f
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !goroutine 1056126 [select]:
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:425 +0x85
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 1054399
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:422 +0x3ce
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !goroutine 1487166 [chan receive]:
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x7173a90, 0xc005c48030})
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:392 +0x94
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !runtime/pprof.Do({0x7173978?, 0xa7318c0?}, {{0xc000da1e60?, 0x1?, 0xc00220c7d0?}}, 0xc00220c7a8)
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !github.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x48c3620?)
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:390 +0x52
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !created by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 1054399
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:382 +0xe5
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !goroutine 1056045 [select]:
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:425 +0x85
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 1054399
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:422 +0x3ce
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !For more context, check log files in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode4288166908
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !****************************************************************************
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !This node experienced a fatal error (printed above), and as a result the
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !process is terminating.
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !will try to determine the root cause, recommend next steps, and we can
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !improve CockroachDB based on your report.
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !Please submit a crash report by following the instructions here:
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !If you would rather not post publicly, please contact us directly at:
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !    [email protected]
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !
F240117 11:17:00.873639 1054399 storage/pebble.go:1621  [n1,s1,pebble] 1 !The Cockroach Labs team appreciates your feedback.

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@arulajmani arulajmani self-assigned this Jan 17, 2024
@arulajmani arulajmani mentioned this issue Jan 17, 2024
Merged
craig bot pushed a commit that referenced this issue Jan 17, 2024
@miraradeva @arulajmani
Merge #117326 #117864
9b10f08 
117326: kvnemesis: add support for non-voting replicas r=kvoli a=miraradeva

This change extends the ChangeReplicas config in kvnemesis to include adding, removing and promoting non-voting replicas, as well as demoting voting replicas.

Fixes: #59060

Release note: None

117864: concurrency: prevent panic in IsKeyLockedByConflictingTxn r=arulajmani a=arulajmani

This was missing a nil check.

Closes #117855
Closes #117854

Release note: None

Co-authored-by: Mira Radeva <[email protected]>
Co-authored-by: Arul Ajmani <[email protected]>
@craig craig bot closed this as completed in 20046d3 Jan 17, 2024
@arulajmani arulajmani added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-kv-transactions Relating to MVCC and the transactional model. labels Jan 17, 2024
@github-project-automation github-project-automation bot moved this to Closed in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arulajmani arulajmani

Labels
A-kv-transactions Relating to MVCC and the transactional model. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team
Projects
No open projects
KV
Status: Closed
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

concurrency: prevent panic in IsKeyLockedByConflictingTxn arulajmani/cockroach
3 participants
@erikgrinaker @arulajmani @cockroach-teamcity