kv: possible batch mutation during commit [inconsistent batch count] #113268

cockroach-sentry · 2023-10-29T06:02:55Z

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4583696564/?referrer=webhooks_plugin

Panic Message:

db.go:811: log.Fatal: pebble: fatal commit error: pebble: inconsistent batch count: 7046526 vs 7046582
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/pebble.(*DB).applyInternal
  | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:811
  | github.com/cockroachdb/pebble.(*DB).Apply
  | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:746
  | github.com/cockroachdb/pebble.(*Batch).Commit
  | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go:1109
  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).Commit
  | 	github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:549
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).ApplyToStateMachine
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:562
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:290
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1005
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:718
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:395
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) secondary error attachment
  | pebble: inconsistent batch count: 7046526 vs 7046582
  | (1) forced error mark
  |   | ×
  |   | github.com/cockroachdb/errors/withstack/*withstack.withStack::
  | Wraps: (2) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
  |   | 	github.com/cockroachdb/pebble/internal/base/external/com_github_cockroachdb_pebble/internal/base/error.go:27
  |   | github.com/cockroachdb/pebble.(*memTable).apply
  |   | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/mem_table.go:224
  |   | github.com/cockroachdb/pebble.(*DB).commitApply
  |   | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:831
  |   | github.com/cockroachdb/pebble.(*commitPipeline).Commit
  |   | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/commit.go:317
  |   | github.com/cockroachdb/pebble.(*DB).applyInternal
  |   | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:808
  |   | github.com/cockroachdb/pebble.(*DB).Apply
  |   | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:746
  |   | github.com/cockroachdb/pebble.(*Batch).Commit
  |   | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go:1109
  |   | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).Commit
  |   | 	github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:549
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).ApplyToStateMachine
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:562
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:290
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1005
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:718
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:395
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
  |   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  |   | runtime.goexit
  |   | 	GOROOT/src/runtime/asm_amd64.s:1594
  | Wraps: (3) pebble: inconsistent batch count: 7046526 vs 7046582
  | Error types: (1) *markers.withMark (2) *withstack.withStack (3) *errutil.leafError
Wraps: (3) log.Fatal: pebble: fatal commit error: pebble: inconsistent batch count: 7046526 vs 7046582
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: pebble: fatal commit error: pebble: inconsistent batch count: 7046526 vs 7046582
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
db.go:811: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595

cockroach/pkg/util/stop/stopper.go

Lines 469 to 471 in 810d4f2

    
           	sp.UpdateGoroutineIDToCurrent() 
        
           	f(ctx) 
        
           }()

https://github.com/cockroachdb/cockroach/blob/810d4f27a7f02b9cc2750cab654ed1c62ac3e75a/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303
https://github.com/cockroachdb/cockroach/blob/810d4f27a7f02b9cc2750cab654ed1c62ac3e75a/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396
https://github.com/cockroachdb/cockroach/blob/810d4f27a7f02b9cc2750cab654ed1c62ac3e75a/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647
https://github.com/cockroachdb/cockroach/blob/810d4f27a7f02b9cc2750cab654ed1c62ac3e75a/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L717-L719
https://github.com/cockroachdb/cockroach/blob/810d4f27a7f02b9cc2750cab654ed1c62ac3e75a/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1004-L1006

cockroach/pkg/kv/kvserver/apply/task.go

Lines 250 to 252 in 810d4f2

    
           for iter.Valid() { 
        
           	if err := t.applyOneBatch(ctx, iter); err != nil { 
        
           		// If the batch threw an error, reject all remaining commands in the

cockroach/pkg/kv/kvserver/apply/task.go

Lines 289 to 291 in 810d4f2

    
           // Apply the persistent state transitions to the state machine. 
        
           if err := batch.ApplyToStateMachine(ctx); err != nil { 
        
           	return err

https://github.com/cockroachdb/cockroach/blob/810d4f27a7f02b9cc2750cab654ed1c62ac3e75a/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go#L561-L563

cockroach/pkg/storage/pebble_batch.go

Lines 548 to 550 in 810d4f2

    
           } 
        
           err := p.batch.Commit(opts) 
        
           if err != nil {

github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go#L1108-L1110
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L745-L747
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L810-L812

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 395
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 646
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 718
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1005
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries at line 251
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).applyOneBatch at line 290
pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go in pkg/kv/kvserver.(*replicaAppBatch).ApplyToStateMachine at line 562
pkg/storage/pebble_batch.go in pkg/storage.(*pebbleBatch).Commit at line 549
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go in github.com/cockroachdb/pebble.(*Batch).Commit at line 1109
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).Apply at line 746
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).applyInternal at line 811

Tags

Tag	Value
Command	start-single-node
Environment	v23.1.2
Go Version	go1.19.4
Platform	linux amd64
Distribution	CCL
Cockroach Release	v23.1.2
Cockroach SHA	`810d4f2`
# of CPUs	6
# of Goroutines	480

Jira issue: CRDB-32838

The text was updated successfully, but these errors were encountered:

jbowens · 2023-10-30T15:50:57Z

I see a few possible explanations here:

The tail of the batch was corrupt due to some form of memory corruption. If we fail to decode something from the batch, we return ok=false from BatchReader.Next and it's indistinguishable from the end of the batch. I filed batch: detect corrupt batches during reading pebble#3023 for erroring appropriately if the batch is corrupt.
The batch received additional write operations after it already entered the commit pipeline. I filed batch: guard against additional batch writes during Commit pebble#3024 for improving detection of write ops against committ[ing|ed] batches.

@cockroachdb/kv – want to check if code auditing reveals some possibility of mutating a batch while it's being committed in this code path? Feel free to close out otherwise, and Storage will address the two filed issues.

Add a guardrail that ensures a batch isn't in the process of being committed when applying a mutation to the Batch. We've observed panics during batch application that suggest concurrent modification of the batch, and these guardrails will help surface any races more readily. Informs cockroachdb/cockroach#113268 Resolve cockroachdb#3024.

Add a guardrail that ensures a batch isn't in the process of being committed when applying a mutation to the Batch. We've observed panics during batch application that suggest concurrent modification of the batch, and these guardrails will help surface any races more readily. Informs cockroachdb/cockroach#113268 Resolve #3024.

blathers-crl · 2023-11-06T15:51:37Z

cc @cockroachdb/replication

erikgrinaker · 2023-11-07T09:24:08Z

@pavelkalinnikov Can you have a look at this when you get a chance?

pav-kv · 2023-11-07T12:42:30Z

@jbowens The batch construction and commit in this code path is sequential and isolated (not reused by other paths), I couldn't quickly find a way in which it could be modified while committing.

Basically it's confined within this block:

cockroach/pkg/kv/kvserver/apply/task.go

Lines 274 to 292 in e23a2ea

    
           batch := t.sm.NewBatch() 
        
           defer batch.Close() 
        
           // Consume a batch-worth of commands. 
        
           pol := trivialPolicy{maxCount: t.batchSize} 
        
           batchIter := takeWhileCmdIter(iter, func(cmd Command) bool { 
        
           	return pol.maybeAdd(cmd.IsTrivial()) 
        
           }) 
        
           // Stage each command in the batch. 
        
           stagedIter, err := mapCmdIter(batchIter, batch.Stage) 
        
           if err != nil { 
        
           	return err 
        
           } 
        
           // Apply the persistent state transitions to the state machine. 
        
           if err := batch.ApplyToStateMachine(ctx); err != nil { 
        
           	return err 
        
           }

pav-kv · 2023-11-07T13:07:15Z

@jbowens Does this error happen only at a single-batch scale? Can it happen that some batch somewhere else is modified while committing, but our batch happens to notice this?

jbowens · 2023-11-07T13:28:00Z

Single batch scale, although batches are pooled, so it could conceivably be the result of a batch elsewhere receiving writes after it’s been closed and reused.

Add a guardrail that ensures a batch isn't in the process of being committed when applying a mutation to the Batch. We've observed panics during batch application that suggest concurrent modification of the batch, and these guardrails will help surface any races more readily. Informs cockroachdb/cockroach#113268 Resolve cockroachdb#3024.

cockroach-sentry added O-sentry Originated from an in-the-wild panic report. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Oct 29, 2023

jbowens changed the title Sentry: db.go:811: log.Fatal: pebble: fatal commit error: pebble: inconsistent batch count: 7046526 vs 7046582 (1) attached stack trace -- stack trace: | github.com/cockroachdb/pebble.(*DB).applyI... storage: pebble: fatal commit error: inconsistent batch count Oct 30, 2023

jbowens added A-storage Relating to our storage engine (Pebble) on-disk storage. T-storage Storage Team labels Oct 30, 2023

This was referenced Oct 30, 2023

batch: detect corrupt batches during reading cockroachdb/pebble#3023

Closed

batch: guard against additional batch writes during Commit cockroachdb/pebble#3024

Closed

blathers-crl bot added the T-kv KV Team label Oct 30, 2023

jbowens added the A-kv Anything in KV that doesn't belong in a more specific category. label Oct 30, 2023

jbowens removed A-storage Relating to our storage engine (Pebble) on-disk storage. T-storage Storage Team labels Oct 30, 2023

jbowens changed the title ~~storage: pebble: fatal commit error: inconsistent batch count~~ kv: possible batch mutation during commit [inconsistent batch count] Oct 30, 2023

jbowens mentioned this issue Oct 30, 2023

db: guard against concurrent batch writes during Commit cockroachdb/pebble#3026

Merged

exalate-issue-sync bot added T-kv-replication and removed T-kv KV Team labels Nov 6, 2023

erikgrinaker assigned pav-kv Nov 7, 2023

pav-kv assigned jbowens Nov 7, 2023

pav-kv closed this as completed Nov 7, 2023

exalate-issue-sync bot unassigned pav-kv Nov 7, 2023

jbowens mentioned this issue Nov 21, 2023

crl-release-23.2: db: guard against concurrent batch writes during Commit cockroachdb/pebble#3093

Merged

github-project-automation bot added this to KV Aug 28, 2024

github-project-automation bot moved this to Closed in KV Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kv: possible batch mutation during commit [inconsistent batch count] #113268

kv: possible batch mutation during commit [inconsistent batch count] #113268

cockroach-sentry commented Oct 29, 2023 •

edited by cockroach-jira-scripts

Loading

jbowens commented Oct 30, 2023 •

edited

Loading

blathers-crl bot commented Nov 6, 2023

erikgrinaker commented Nov 7, 2023

pav-kv commented Nov 7, 2023 •

edited

Loading

pav-kv commented Nov 7, 2023

jbowens commented Nov 7, 2023

kv: possible batch mutation during commit [inconsistent batch count] #113268

kv: possible batch mutation during commit [inconsistent batch count] #113268

Comments

cockroach-sentry commented Oct 29, 2023 • edited by cockroach-jira-scripts Loading

Tags

jbowens commented Oct 30, 2023 • edited Loading

blathers-crl bot commented Nov 6, 2023

erikgrinaker commented Nov 7, 2023

pav-kv commented Nov 7, 2023 • edited Loading

pav-kv commented Nov 7, 2023

jbowens commented Nov 7, 2023

cockroach-sentry commented Oct 29, 2023 •

edited by cockroach-jira-scripts

Loading

jbowens commented Oct 30, 2023 •

edited

Loading

pav-kv commented Nov 7, 2023 •

edited

Loading