kvserver: span assertions fire in replication layer

[tbg]

Describe the problem

See the detailed analysis here.

We need to make structural improvements to the span lifecycle (document, simplify, introduce invariant checks, testing, etc) that avoid this class of bugs.

See also #86547 and all the other work in CRDB-25287 (which is related and made important simplifications, but didn't focus on the span lifecycles in particular).

To Reproduce

Unclear

Jira issue: CRDB-30092

[blathers-crl]

cc @cockroachdb/replication

[blathers-crl]

Hi @tbg, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[pav-kv]

Failures we've collected so far.

panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
	panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

While applying entries:

• kv/kvnemesis: TestKVNemesisMultiNode failed #101721 (comment) [accidental encounter, dup of TestKVNemesisMultiNode below]
• jobs: "system.job_info does not exist" during cluster upgrade #103239 (comment) [another accidental repro]
• roachtest: tpccbench/nodes=3/cpu=16/mt-shared-process failed [replication span assertion; needs #107521] #107242
• roachtest: weekly/tpcc/headroom failed #107853
• kv/kvnemesis: TestKVNemesisMultiNode failed #108241

While flushing proposals:

• kvserver: use of span after finish #108534 (with repro steps)
• ccl/multiregionccl: TestMultiRegionDataDriven failed #98020 (comment) [consistently fails]

[lidorcarmel]

@pavelkalinnikov looks like this is another one #107935 (comment) in case it helps you..