sentry: replica_proposal.go:441: log.Fatal: checksum for AddSSTable at index term 6, index 26 does not match; at proposal time f3d30345 (4090692421), now f874a246 (4168393286) (1) attached stack trace -- st...

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3714172783/?referrer=webhooks_plugin

Panic message:

replica_proposal.go:441: log.Fatal: checksum for AddSSTable at index term 6, index 26 does not match; at proposal time f3d30345 (4090692421), now f874a246 (4168393286)
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.addSSTablePreApply
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:441
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).runPreApplyTriggersAfterStagingWriteBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:647
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:537
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:184
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:280
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:247
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1028
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:648
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:512
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:308
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:494
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (2) log.Fatal: checksum for AddSSTable at index term 6, index 26 does not match; at proposal time f3d30345 (4090692421), now f874a246 (4168393286)
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: checksum for AddSSTable at index term 6, index 26 does not match; at proposal time f3d30345 (4090692421), now f874a246 (4168393286)
replica_proposal.go:441: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

https://github.com/cockroachdb/cockroach/blob/477af1d876e3e62b26900854c2f33eaa7cec73db/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L440-L442 in pkg/kv/kvserver.addSSTablePreApply
https://github.com/cockroachdb/cockroach/blob/477af1d876e3e62b26900854c2f33eaa7cec73db/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L646-L648 in pkg/kv/kvserver.(*replicaAppBatch).runPreApplyTriggersAfterStagingWriteBatch
https://github.com/cockroachdb/cockroach/blob/477af1d876e3e62b26900854c2f33eaa7cec73db/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L536-L538 in pkg/kv/kvserver.(*replicaAppBatch).Stage

cockroach/pkg/kv/kvserver/apply/cmd.go

Lines 183 to 185 in 477af1d

	cur := iter.Cur()
	checked, err := fn(cur.Ctx(), cur)
	if err != nil {

in pkg/kv/kvserver/apply.mapCmdIter

cockroach/pkg/kv/kvserver/apply/task.go

Lines 279 to 281 in 477af1d

	// Stage each command in the batch.
	stagedIter, err := mapCmdIter(batchIter, batch.Stage)
	if err != nil {

in pkg/kv/kvserver/apply.(*Task).applyOneBatch

cockroach/pkg/kv/kvserver/apply/task.go

Lines 246 to 248 in 477af1d

	for iter.Valid() {
	if err := t.applyOneBatch(ctx, iter); err != nil {
	// If the batch threw an error, reject all remaining commands in the

in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
https://github.com/cockroachdb/cockroach/blob/477af1d876e3e62b26900854c2f33eaa7cec73db/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1027-L1029 in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
https://github.com/cockroachdb/cockroach/blob/477af1d876e3e62b26900854c2f33eaa7cec73db/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L647-L649 in pkg/kv/kvserver.(*Replica).handleRaftReady
https://github.com/cockroachdb/cockroach/blob/477af1d876e3e62b26900854c2f33eaa7cec73db/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L511-L513 in pkg/kv/kvserver.(*Store).processReady
https://github.com/cockroachdb/cockroach/blob/477af1d876e3e62b26900854c2f33eaa7cec73db/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L307-L309 in pkg/kv/kvserver.(*raftScheduler).worker

cockroach/pkg/util/stop/stopper.go

Lines 493 to 495 in 477af1d

	f(ctx)
	}()

in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
GOROOT/src/runtime/asm_amd64.s#L1580-L1582 in runtime.goexit

pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.addSSTablePreApply at line 441
pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaAppBatch).runPreApplyTriggersAfterStagingWriteBatch at line 647
pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaAppBatch).Stage at line 537
pkg/kv/kvserver/apply/cmd.go in pkg/kv/kvserver/apply.mapCmdIter at line 184
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).applyOneBatch at line 280
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries at line 247
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1028
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 648
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 512
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).worker at line 308
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 494
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1581

Tag	Value
Cockroach Release	v22.1.10
Cockroach SHA:	477af1d
Platform	linux amd64
Distribution	CCL
Environment	v22.1.10
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-21130

[yuzefovich]

dup of #90834