roachtest: c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed - span reuse after finish

[cockroach-teamcity]

roachtest.c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed with artifacts on master @ e0de1f12d4f496fc7e4050a33e96f8c635c7a27c:

(assertions.go:333).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:917
	            				main/pkg/cmd/roachtest/monitor.go:119
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
	            				GOROOT/src/runtime/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	read tcp 172.17.0.3:37308->34.138.79.86:26257: read: connection reset by peer
	Test:       	c2c/tpcc/warehouses=1000/duration=60/cutover=30
(require.go:1360).NoError: FailNow called
(test_runner.go:1146).func1: 1 dead node(s) detected
test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=1000/duration=60/cutover=30/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed #108699 roachtest: c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-23.1]

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-30659

[stevendanna]

* use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | [...repeated from below...]
* Wraps: (2) assertion failure
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442
*   | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450
*   | github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog
*   | 	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92
*   | github.com/cockroachdb/cockroach/pkg/util/log.vEventf
*   | 	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:212
*   | github.com/cockroachdb/cockroach/pkg/util/log.VEvent
*   | 	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:231
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:601
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:775
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484
*   | runtime.goexit
*   | 	GOROOT/src/runtime/asm_amd64.s:1594
* Wraps: (4) use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*

which looks like: #108767

[stevendanna]

Before that kills the node, we see a lot of batch requests failing with context cancelled:

I230816 10:04:49.879412 71158 server/node.go:1373 ⋮ [T1,n4] 1129  batch request ‹QueryTxn [/Tenant/2/Table/110/1/373/1/-2102/0,/Min)› failed with error: context canceled
I230816 10:04:49.879412 71158 server/node.go:1373 â‹® [T1,n4] 1129 +trace:
I230816 10:04:49.879412 71158 server/node.go:1373 â‹® [T1,n4] 1129 +     0.000ms      0.000ms    === operation:/cockroach.roachpb.Internal/Batch
I230816 10:04:49.879509 71159 server/node.go:1373 ⋮ [T1,n4] 1130  batch request ‹QueryTxn [/Tenant/2/Table/110/1/373/1/-2102/0,/Min)› failed with error: context canceled
I230816 10:04:49.879509 71159 server/node.go:1373 â‹® [T1,n4] 1130 +trace:
I230816 10:04:49.879509 71159 server/node.go:1373 â‹® [T1,n4] 1130 +     0.000ms      0.000ms    === operation:/cockroach.roachpb.Internal/Batch
I230816 10:04:49.882649 70702 server/node.go:1373 ⋮ [T1,n4] 1131  batch request ‹QueryTxn [/Tenant/2/Table/110/1/387/1/-2102/0,/Min)› failed with error: context canceled

and this starts right after the initial scan is likely starting up from the replication stream...

[msbutler]

should be fixed after #108775 lands

[msbutler]

@pavelkalinnikov I'm reassigning this to you as your patch #108775 will fix this.

[blathers-crl]

cc @cockroachdb/replication