kvserver: v21.2.9: not using applied state key in v21.1

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3398694605/?referrer=webhooks_plugin

Panic message:

replica_proposal.go:898: log.Fatal: not using applied state key in v21.1
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:898
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:933
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:110
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:156
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:377
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:107
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:43
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*pendingLeaseRequest).requestLeaseAsync.func2
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_range_lease.go:427
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:442
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) log.Fatal: not using applied state key in v21.1
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: not using applied state key in v21.1
replica_proposal.go:898: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

cockroach/pkg/kv/kvserver/replica_proposal.go

Lines 897 to 899 in 11787ed

	if migrationVersion.Less(activeVersion) {
	log.Fatal(ctx, "not using applied state key in v21.1")
	}

in pkg/kv/kvserver.(*Replica).evaluateProposal

cockroach/pkg/kv/kvserver/replica_proposal.go

Lines 932 to 934 in 11787ed

	) (*ProposalData, *roachpb.Error) {
	res, needConsensus, pErr := r.evaluateProposal(ctx, idKey, ba, lul, latchSpans, lockSpans)

in pkg/kv/kvserver.(*Replica).requestToProposal

cockroach/pkg/kv/kvserver/replica_raft.go

Lines 109 to 111 in 11787ed

	idKey := makeIDKey()
	proposal, pErr := r.requestToProposal(ctx, idKey, ba, st, lul, g.LatchSpans(), g.LockSpans())
	log.Event(proposal.ctx, "evaluated request")

in pkg/kv/kvserver.(*Replica).evalAndPropose

cockroach/pkg/kv/kvserver/replica_write.go

Lines 155 to 157 in 11787ed

	// evalAndPropose.
	ch, abandon, _, pErr := r.evalAndPropose(ctx, ba, g, st, localUncertaintyLimit, tok.Move(ctx))
	if pErr != nil {

in pkg/kv/kvserver.(*Replica).executeWriteBatch

cockroach/pkg/kv/kvserver/replica_send.go

Lines 376 to 378 in 11787ed

	br, g, pErr = fn(r, ctx, ba, g)
	if pErr == nil {

in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries

cockroach/pkg/kv/kvserver/replica_send.go

Lines 106 to 108 in 11787ed

	fn := (*Replica).executeWriteBatch
	br, pErr = r.executeBatchWithConcurrencyRetries(ctx, ba, fn)
	} else if ba.IsAdmin() {

in pkg/kv/kvserver.(*Replica).sendWithRangeID

cockroach/pkg/kv/kvserver/replica_send.go

Lines 42 to 44 in 11787ed

	) (*roachpb.BatchResponse, *roachpb.Error) {
	return r.sendWithRangeID(ctx, r.RangeID, &ba)
	}

in pkg/kv/kvserver.(*Replica).Send

cockroach/pkg/kv/kvserver/replica_range_lease.go

Lines 426 to 428 in 11787ed

	ba.Add(leaseReq)
	_, pErr = p.repl.Send(ctx, ba)
	}

in pkg/kv/kvserver.(*pendingLeaseRequest).requestLeaseAsync.func2

cockroach/pkg/util/stop/stopper.go

Lines 441 to 443 in 11787ed

	f(ctx)
	}()

in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
/usr/local/go/src/runtime/asm_amd64.s#L1370-L1372 in runtime.goexit

pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).evaluateProposal at line 898
pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).requestToProposal at line 933
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).evalAndPropose at line 110
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 156
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 377
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).sendWithRangeID at line 107
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).Send at line 43
pkg/kv/kvserver/replica_range_lease.go in pkg/kv/kvserver.(*pendingLeaseRequest).requestLeaseAsync.func2 at line 427
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 442
/usr/local/go/src/runtime/asm_amd64.s in runtime.goexit at line 1371

Tag	Value
Cockroach Release	v21.2.9
Cockroach SHA:	11787ed
Platform	linux amd64
Distribution	CCL
Environment	v21.2.9
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-17245

[nvanbenschoten]

We see that this fatal was hit all on the same cluster and that this is the same failure mode we saw in #62267. That issue was fixed by #62838.