sentry: store_raft.go:517: log.Fatal: ???: raft closed timestamp regression in cmd: "\x1b\x1c\x17\xa6E\x0e\xa8\xaf" (term: 8, index: 6670); batch state: 1653485974.276224000,0, command: 1653485974.232592000,0...

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3300816983/?referrer=webhooks_plugin

Panic message:

store_raft.go:517: log.Fatal: ???: raft closed timestamp regression in cmd: "\x1b\x1c\x17\xa6E\x0e\xa8\xaf" (term: 8, index: 6670); batch state: 1653485974.276224000,0, command: 1653485974.232592000,0, lease: repl=(n1,s1):1 seq=1 start=0,0 exp=1653485986.232731000,0 pro=1653485977.232731000,0, req: EndTxn(commit:false) [×], [txn: bb3e5e72], [can-forward-ts], applying at LAI: 2512.
(1) assertion failure
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).assertNoCmdClosedTimestampRegression
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_application_state_machine.go:1101
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_application_state_machine.go:465
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:175
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:280
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:247
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:824
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:487
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:516
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker
| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:284
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:446
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1374
Wraps: (3) raft closed timestamp regression in cmd: "\x1b\x1c\x17\xa6E\x0e\xa8\xaf" (term: 8, index: 6670); batch state: 1653485974.276224000,0, command: 1653485974.232592000,0, lease: repl=(n1,s1):1 seq=1 start=0,0 exp=1653485986.232731000,0 pro=1653485977.232731000,0, req: EndTxn(commit:false) [×], [txn: bb3e5e72], [can-forward-ts], applying at LAI: 2512.
| Closed timestamp was set by req: <unknown; not leaseholder or not lease request> under lease: %!s(PANIC=SafeFormatter method: value method github.com/cockroachdb/cockroach/pkg/roachpb.Lease.SafeFormat called using nil *Lease pointer); applied at LAI: 0. Batch idx: 0.
| This assertion will fire again on restart; to ignore run with env var COCKROACH_RAFT_CLOSEDTS_ASSERTIONS_ENABLED=false
| Raft log tail:
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
|
| ×
|
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
|
| ×
| ×
| ×
|
| ×
| ×
| ×
| ×
Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
--
*errutil.leafError: log.Fatal: ???: raft closed timestamp regression in cmd: "\x1b\x1c\x17\xa6E\x0e\xa8\xaf" (term: 8, index: 6670); batch state: 1653485974.276224000,0, command: 1653485974.232592000,0, lease: repl=(n1,s1):1 seq=1 start=0,0 exp=1653485986.232731000,0 pro=1653485977.232731000,0, req: EndTxn(commit:false) [×], [txn: bb3e5e72], [can-forward-ts], applying at LAI: 2512. (1)
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack::: (2)
store_raft.go:517: *withstack.withStack (top exception)
(check the extra data payloads)

Stacktrace (expand for inline code snippets):

cockroach/pkg/kv/kvserver/store_raft.go

Lines 516 to 518 in bf2bef1

	stats, expl, err := r.handleRaftReady(ctx, noSnap)
	removed := maybeFatalOnRaftReadyErr(ctx, expl, err)
	elapsed := timeutil.Since(start)

in pkg/kv/kvserver.(*Store).processReady

cockroach/pkg/kv/kvserver/scheduler.go

Lines 283 to 285 in bf2bef1

	if state.flags&stateRaftReady != 0 {
	s.processor.processReady(ctx, id)
	}

in pkg/kv/kvserver.(*raftScheduler).worker

cockroach/pkg/util/stop/stopper.go

Lines 445 to 447 in bf2bef1

	f(ctx)
	}()

in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
/usr/local/go/src/runtime/asm_amd64.s#L1373-L1375 in runtime.goexit

pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 517
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).worker at line 284
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 446
/usr/local/go/src/runtime/asm_amd64.s in runtime.goexit at line 1374

Tag	Value
Cockroach Release	v21.1.19
Cockroach SHA:	bf2bef1
Platform	darwin amd64
Distribution	CCL
Environment	v21.1.19
Command	start-single-node
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-16442

[yuzefovich]

dup of #70894