kv/kvserver: TestTxnClearRangeIntents failed #64088

cockroach-teamcity · 2021-04-22T17:41:05Z

kv/kvserver.TestTxnClearRangeIntents failed with artifacts on master @ ee8b38ef3e0f7a41ae277c127bb9db5989ef47d0:

        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:294 +0x3a5
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).CheckAllowed(0xc002674d20, 0x0, 0xc0025fe700, 0x8, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:225 +0xba
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*EngineIterator).SeekEngineKeyGE(0xc000f24c60, 0xc0025fe700, 0x8, 0x8, 0x0, 0x0, 0x0, 0x7, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/batch.go:256 +0x1a8
        	            	github.com/cockroachdb/cockroach/pkg/storage.ScanSeparatedIntents(0x86209c0, 0xc001f301b0, 0xc0034b2872, 0x2, 0x2, 0xc0034b2874, 0x1, 0x1, 0x3e8, 0xf4240, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine.go:888 +0x385
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.ClearRange(0x85b3400, 0xc00328ec30, 0x7fe871f78c70, 0xc001f301b0, 0x8715ae0, 0xc002ad2540, 0x16783a9e808718ad, 0x0, 0x100000001, 0x1, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_clear_range.go:96 +0x30a
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand(0x85b3400, 0xc00328ec30, 0xc0025fe6d8, 0x8, 0x0, 0x7fe871f78c70, 0xc001f301b0, 0x8715ae0, 0xc002ad2540, 0xc000040280, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:506 +0x319
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch(0x85b3400, 0xc00328ec30, 0xc0025fe6d8, 0x8, 0x7fe871f78c70, 0xc001f301b0, 0x8715ae0, 0xc002ad2540, 0xc000040280, 0xc001447360, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:271 +0x4bd
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper(0xc002b82400, 0x85b3400, 0xc00328ec30, 0xc0025fe6d8, 0x8, 0x8715ae0, 0xc002ad2540, 0xc000040280, 0xc001447360, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:657 +0x19d
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes(0xc002b82400, 0x85b3400, 0xc00328ec30, 0xc0025fe6d8, 0x8, 0x8715ae0, 0xc002ad2540, 0xc000040280, 0xc001447360, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:625 +0x1e5
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch(0xc002b82400, 0x85b3400, 0xc00328ec30, 0xc0025fe6d8, 0x8, 0xc001447360, 0x0, 0x0, 0xc002674d20, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:440 +0x276
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal(0xc002b82400, 0x85b3400, 0xc00328ec30, 0xc0025fe6d8, 0x8, 0xc001447360, 0x0, 0x0, 0xc002674d20, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:775 +0x177
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal(0xc002b82400, 0x85b3400, 0xc00328ec30, 0xc0025fe6d8, 0x8, 0xc001447360, 0x0, 0x0, 0xc0034b2b70, 0x100000001, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:906 +0xcc
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose(0xc002b82400, 0x85b3400, 0xc00328ec30, 0xc001447360, 0xc002ad20e0, 0x0, 0x0, 0xc0034b2b70, 0x100000001, 0x1, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:84 +0x20e
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc002b82400, 0x85b3400, 0xc00328ec30, 0xc001447360, 0xc002ad20e0, 0x0, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:144 +0x8a5
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc002b82400, 0x85b3400, 0xc00328ec30, 0xc001447360, 0x70ef010, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:275 +0x409
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc002b82400, 0x85b3400, 0xc002a7eb10, 0x5, 0xc001447360, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:95 +0x629
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:34
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0006d0800, 0x85b3380, 0xc00019c010, 0x16783a9e808718ad, 0x0, 0x100000001, 0x1, 0x0, 0x5, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:195 +0x573
        	            	github.com/cockroachdb/cockroach/pkg/kv.Wrap.func1(0x85b3380, 0xc00019c010, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:456 +0x175
        	            	github.com/cockroachdb/cockroach/pkg/kv.SenderFunc.Send(0xc000f24b80, 0x85b3380, 0xc00019c010, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:392 +0xcb
        	            	github.com/cockroachdb/cockroach/pkg/kv.SendWrappedWith(0x85b3380, 0xc00019c010, 0x851f7c0, 0xc000f24b80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:432 +0x195
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.TestTxnClearRangeIntents(0xc001152180)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/txn_recovery_integration_test.go:398 +0x1b39
        	            	testing.tRunner(0xc001152180, 0x70f0428)
        	            		/usr/local/go/src/testing/testing.go:1123 +0x203
        	            	created by testing.(*T).Run
        	            		/usr/local/go/src/testing/testing.go:1168 +0x5bc
    panic.go:636: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTxnClearRangeIntents465701679
--- FAIL: TestTxnClearRangeIntents (0.19s)

Reproduce

To reproduce, try:

make stressrace TESTS=TestTxnClearRangeIntents PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

GOFLAGS=-json

/cc @erikgrinaker _{This test on roachdash | Improve this report!}

The text was updated successfully, but these errors were encountered:

Refs: cockroachdb#64088 Reason: flaky test Generated by bin/skip-test. Release justification: non-production code changes Release note: None

64090: kv/kvserver: skip TestTxnClearRangeIntents under race r=sumeerbhola a=sumeerbhola Refs: #64088 Reason: flaky test Generated by bin/skip-test. Release justification: non-production code changes Release note: None 64091: kv/kvserver: skip TestMergeQueue r=sumeerbhola a=sumeerbhola Refs: #64056 Reason: flaky test Generated by bin/skip-test. Release justification: non-production code changes Release note: None Co-authored-by: sumeerbhola <[email protected]>

cockroach-teamcity · 2021-04-22T21:10:48Z

kv/kvserver.TestTxnClearRangeIntents failed with artifacts on master @ 97955d46cef43897f0281b336e521bc89a5fa400:

        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:294 +0x3a5
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).CheckAllowed(0xc00108daa0, 0x0, 0xc0014ff0e8, 0x8, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:225 +0xba
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*EngineIterator).SeekEngineKeyGE(0xc0011e5300, 0xc0014ff0e8, 0x8, 0x8, 0x0, 0x0, 0x0, 0x7, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/batch.go:256 +0x1a8
        	            	github.com/cockroachdb/cockroach/pkg/storage.ScanSeparatedIntents(0x8621fa0, 0xc002dcb7a0, 0xc002eda142, 0x2, 0x2, 0xc002eda144, 0x1, 0x1, 0x3e8, 0xf4240, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine.go:888 +0x385
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.ClearRange(0x85b49e0, 0xc00303df80, 0x7ff85e21e378, 0xc002dcb7a0, 0x87170c0, 0xc002294cb0, 0x167845504ce20b35, 0x0, 0x100000001, 0x1, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_clear_range.go:96 +0x30a
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand(0x85b49e0, 0xc00303df80, 0xc0014ff0e0, 0x8, 0x0, 0x7ff85e21e378, 0xc002dcb7a0, 0x87170c0, 0xc002294cb0, 0xc002538300, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:506 +0x319
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch(0x85b49e0, 0xc00303df80, 0xc0014ff0e0, 0x8, 0x7ff85e21e378, 0xc002dcb7a0, 0x87170c0, 0xc002294cb0, 0xc002538300, 0xc00067fd60, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:271 +0x4bd
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper(0xc00206f200, 0x85b49e0, 0xc00303df80, 0xc0014ff0e0, 0x8, 0x87170c0, 0xc002294cb0, 0xc002538300, 0xc00067fd60, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:657 +0x19d
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes(0xc00206f200, 0x85b49e0, 0xc00303df80, 0xc0014ff0e0, 0x8, 0x87170c0, 0xc002294cb0, 0xc002538300, 0xc00067fd60, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:625 +0x1e5
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch(0xc00206f200, 0x85b49e0, 0xc00303df80, 0xc0014ff0e0, 0x8, 0xc00067fd60, 0x0, 0x0, 0xc00108daa0, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:440 +0x276
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal(0xc00206f200, 0x85b49e0, 0xc00303df80, 0xc0014ff0e0, 0x8, 0xc00067fd60, 0x0, 0x0, 0xc00108daa0, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:782 +0x177
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal(0xc00206f200, 0x85b49e0, 0xc00303df80, 0xc0014ff0e0, 0x8, 0xc00067fd60, 0x0, 0x0, 0xc002be6740, 0x100000001, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:913 +0xcc
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose(0xc00206f200, 0x85b49e0, 0xc00303df80, 0xc00067fd60, 0xc002294c40, 0x0, 0x0, 0xc002be6740, 0x100000001, 0x1, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:84 +0x20e
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc00206f200, 0x85b49e0, 0xc00303df80, 0xc00067fd60, 0xc002294c40, 0x0, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:144 +0x8a5
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00206f200, 0x85b49e0, 0xc00303df80, 0xc00067fd60, 0x70f0178, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:275 +0x409
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00206f200, 0x85b49e0, 0xc000217890, 0x5, 0xc00067fd60, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:95 +0x629
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:34
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0010fd800, 0x85b4960, 0xc000074140, 0x167845504ce20b35, 0x0, 0x100000001, 0x1, 0x0, 0x5, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:195 +0x573
        	            	github.com/cockroachdb/cockroach/pkg/kv.Wrap.func1(0x85b4960, 0xc000074140, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:456 +0x175
        	            	github.com/cockroachdb/cockroach/pkg/kv.SenderFunc.Send(0xc0011e5240, 0x85b4960, 0xc000074140, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:392 +0xcb
        	            	github.com/cockroachdb/cockroach/pkg/kv.SendWrappedWith(0x85b4960, 0xc000074140, 0x8520d60, 0xc0011e5240, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:432 +0x195
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.TestTxnClearRangeIntents(0xc00143c180)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/txn_recovery_integration_test.go:398 +0x1b39
        	            	testing.tRunner(0xc00143c180, 0x70f1590)
        	            		/usr/local/go/src/testing/testing.go:1123 +0x203
        	            	created by testing.(*T).Run
        	            		/usr/local/go/src/testing/testing.go:1168 +0x5bc
    panic.go:636: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTxnClearRangeIntents846422537
--- FAIL: TestTxnClearRangeIntents (0.20s)

Reproduce

To reproduce, try:

make stressrace TESTS=TestTxnClearRangeIntents PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

GOFLAGS=-json

/cc @erikgrinaker _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-04-22T21:40:56Z

kv/kvserver.TestTxnClearRangeIntents failed with artifacts on master @ ea6dc019d083434e0df6b3181efc7522405f79b6:

        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:294 +0x3a5
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).CheckAllowed(0xc002cc4d80, 0x0, 0xc002bfc4d0, 0x8, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:225 +0xba
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*EngineIterator).SeekEngineKeyGE(0xc002466860, 0xc002bfc4d0, 0x8, 0x8, 0x0, 0x0, 0x0, 0x7, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/batch.go:256 +0x1a8
        	            	github.com/cockroachdb/cockroach/pkg/storage.ScanSeparatedIntents(0x8621fa0, 0xc002834360, 0xc001d34a62, 0x2, 0x2, 0xc001d34a64, 0x1, 0x1, 0x3e8, 0xf4240, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine.go:888 +0x385
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.ClearRange(0x85b49e0, 0xc003611fb0, 0x7ff80d3902f0, 0xc002834360, 0x87170c0, 0xc002b405b0, 0x167847c575a7b3da, 0x0, 0x100000001, 0x1, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_clear_range.go:96 +0x30a
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand(0x85b49e0, 0xc003611fb0, 0xc002bfc4b8, 0x8, 0x0, 0x7ff80d3902f0, 0xc002834360, 0x87170c0, 0xc002b405b0, 0xc000022380, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:506 +0x319
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch(0x85b49e0, 0xc003611fb0, 0xc002bfc4b8, 0x8, 0x7ff80d3902f0, 0xc002834360, 0x87170c0, 0xc002b405b0, 0xc000022380, 0xc001719720, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:271 +0x4bd
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper(0xc002047200, 0x85b49e0, 0xc003611fb0, 0xc002bfc4b8, 0x8, 0x87170c0, 0xc002b405b0, 0xc000022380, 0xc001719720, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:657 +0x19d
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes(0xc002047200, 0x85b49e0, 0xc003611fb0, 0xc002bfc4b8, 0x8, 0x87170c0, 0xc002b405b0, 0xc000022380, 0xc001719720, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:625 +0x1e5
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch(0xc002047200, 0x85b49e0, 0xc003611fb0, 0xc002bfc4b8, 0x8, 0xc001719720, 0x0, 0x0, 0xc002cc4d80, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:440 +0x276
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal(0xc002047200, 0x85b49e0, 0xc003611fb0, 0xc002bfc4b8, 0x8, 0xc001719720, 0x0, 0x0, 0xc002cc4d80, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:782 +0x177
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal(0xc002047200, 0x85b49e0, 0xc003611fb0, 0xc002bfc4b8, 0x8, 0xc001719720, 0x0, 0x0, 0xc001d34da0, 0x100000001, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:913 +0xcc
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose(0xc002047200, 0x85b49e0, 0xc003611fb0, 0xc001719720, 0xc002b400e0, 0x0, 0x0, 0xc001d34da0, 0x100000001, 0x1, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:84 +0x20e
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc002047200, 0x85b49e0, 0xc003611fb0, 0xc001719720, 0xc002b400e0, 0x0, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:144 +0x8a5
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc002047200, 0x85b49e0, 0xc003611fb0, 0xc001719720, 0x70f0178, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:275 +0x409
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc002047200, 0x85b49e0, 0xc00277ddd0, 0x5, 0xc001719720, 0x0, 0x0)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:95 +0x629
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:34
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc001245800, 0x85b4960, 0xc000126010, 0x167847c575a7b3da, 0x0, 0x100000001, 0x1, 0x0, 0x5, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:195 +0x573
        	            	github.com/cockroachdb/cockroach/pkg/kv.Wrap.func1(0x85b4960, 0xc000126010, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:456 +0x175
        	            	github.com/cockroachdb/cockroach/pkg/kv.SenderFunc.Send(0xc002466760, 0x85b4960, 0xc000126010, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:392 +0xcb
        	            	github.com/cockroachdb/cockroach/pkg/kv.SendWrappedWith(0x85b4960, 0xc000126010, 0x8520d60, 0xc002466760, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/sender.go:432 +0x195
        	            	github.com/cockroachdb/cockroach/pkg/kv/kvserver.TestTxnClearRangeIntents(0xc0027e2180)
        	            		/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/txn_recovery_integration_test.go:398 +0x1b39
        	            	testing.tRunner(0xc0027e2180, 0x70f1590)
        	            		/usr/local/go/src/testing/testing.go:1123 +0x203
        	            	created by testing.(*T).Run
        	            		/usr/local/go/src/testing/testing.go:1168 +0x5bc
    panic.go:636: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTxnClearRangeIntents433252767
--- FAIL: TestTxnClearRangeIntents (0.19s)

Reproduce

To reproduce, try:

make stressrace TESTS=TestTxnClearRangeIntents PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

GOFLAGS=-json

/cc @erikgrinaker _{This test on roachdash | Improve this report!}

erikgrinaker · 2021-04-23T19:56:23Z

This fails because race builds enable spanset assertions:

cockroach/pkg/kv/kvserver/replica_eval_context.go

Lines 39 to 44 in bc52ebf

    
           if util.RaceEnabled { 
        
           	return &SpanSetReplicaEvalContext{ 
        
           		i:  r, 
        
           		ss: *ss, 
        
           	} 
        
           }

Under -race, when separated intents are randomly enabled for tests, ClearRange() will attempt to scan the lock table but fails the assertion because it hasn't declared the lock table keys in the span.

        	Test:       	TestTxnClearRangeIntents
        	Messages:   	error: cannot read undeclared span /Local/Lock/Intent"aa"
        	            	declared:
        	            	read local: /Local/Range/System/"tse"/RangeDescriptor at 0,0
        	            	write global: {aa-x} at 1619207002.024321000,0

@sumeerbhola A naïve solution is to simply add the intent keys to the span declarations for ClearRange, does that make sense? Asking since intents are themselves locks, so I'm not sure if this is something the lock manager (or DefaultDeclareIsolatedKeys) should handle automatically instead.

cockroach/pkg/kv/kvserver/batcheval/cmd_clear_range.go

Lines 38 to 48 in 8f8c02a

    
           func declareKeysClearRange( 
        
           	rs ImmutableRangeState, 
        
           	header roachpb.Header, 
        
           	req roachpb.Request, 
        
           	latchSpans, lockSpans *spanset.SpanSet, 
        
           ) { 
        
           	DefaultDeclareIsolatedKeys(rs, header, req, latchSpans, lockSpans) 
        
           	// We look up the range descriptor key to check whether the span 
        
           	// is equal to the entire range for fast stats updating. 
        
           	latchSpans.AddNonMVCC(spanset.SpanReadOnly, roachpb.Span{Key: keys.RangeDescriptorKey(rs.GetStartKey())}) 
        
           }

erikgrinaker · 2021-05-03T15:08:34Z

@sumeerbhola Just a reminder about this, would appreciate your input.

sumeerbhola · 2021-05-03T18:18:56Z

thanks for pinging -- I somehow missed this.

You are right that DefaultDeclareIsolatedKeys ought to be sufficient, and we are seeing no complaints when using the MVCCIterator and only seeing it on EngineIterator. I think the problem is that I never setup the code in kvserver/spanset/batch.go to correctly do span checking for an EngineIterator.

This doesn't occur in the usual use of EngineIterator as an implementation detail of intentInterleavingIter. The spanSetReader.NewMVCCIterator calls into the real implementation and the actual Reader implementation (pebbleReadOnly, pebbleBatch) uses wrapReader to wrap itself. This means NewEngineIterator is called on this actual Reader implementation and does not get wrapped in spanset.EngineIterator. The resulting intentInterleavingIter does get wrapped in spanset.MVCCIterator so those calls do get checked (but not the internal implementation details of intentInterleavingIter). I think this partial checking is fine for the intentInterleavingIter case.

The difficulty in fixing this properly is that in general someone constructing an EngineIterator is not necessarily doing it only to look at separated locks. If we knew it was separated locks, we could, in the first call to spanSetReader.NewEngineIterator, translate the SpanSet into another SpanSet for the lock table. This would use keys.LockTableSingleKey in a manner similar to what newIntentInterleavingIter does with translating bounds. Maybe this is ok given the limited purpose of spanSetReader -- we are constructing these in one place in newBatchedEngine with the spans equal to the latchSpans. To be more thorough, one could additionally pass the lockSpans and use that to initiate the translation.
Or we could ignore this and strip out the checking in spanset.EngineIterator, and leave a TODO -- I'm slightly worried about this option since fixing the above checking should be straightforward could save us a lot of grief with bugs in the future.

erikgrinaker · 2021-05-13T11:57:26Z

Thanks for the detailed explanation @sumeerbhola, I submitted a draft in #65124 (without any tests or perf considerations). Is this roughly what you had in mind?

cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 22, 2021

cockroach-teamcity added this to the 21.1 milestone Apr 22, 2021

sumeerbhola mentioned this issue Apr 22, 2021

kv/kvserver: skip TestTxnClearRangeIntents under race #64090

Merged

erikgrinaker self-assigned this Apr 22, 2021

erikgrinaker mentioned this issue May 13, 2021

kvserver: add lock table spans to spanset assertions #65124

Merged

craig bot closed this as completed in 08322ff May 18, 2021

erikgrinaker mentioned this issue May 23, 2021

release-21.1: kvserver: add lock table spans to spanset assertions #65593

Merged

erikgrinaker mentioned this issue Oct 3, 2021

kvserver: QueryResolvedTimestamp should declare lock table keys #71053

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kv/kvserver: TestTxnClearRangeIntents failed #64088

kv/kvserver: TestTxnClearRangeIntents failed #64088

cockroach-teamcity commented Apr 22, 2021

cockroach-teamcity commented Apr 22, 2021

cockroach-teamcity commented Apr 22, 2021

erikgrinaker commented Apr 23, 2021 •

edited

Loading

erikgrinaker commented May 3, 2021

sumeerbhola commented May 3, 2021

erikgrinaker commented May 13, 2021

kv/kvserver: TestTxnClearRangeIntents failed #64088

kv/kvserver: TestTxnClearRangeIntents failed #64088

Comments

cockroach-teamcity commented Apr 22, 2021

cockroach-teamcity commented Apr 22, 2021

cockroach-teamcity commented Apr 22, 2021

erikgrinaker commented Apr 23, 2021 • edited Loading

erikgrinaker commented May 3, 2021

sumeerbhola commented May 3, 2021

erikgrinaker commented May 13, 2021

erikgrinaker commented Apr 23, 2021 •

edited

Loading