Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pebble: v20.2.4: index out of range (looks like an integer overflow) #69906

Closed
cockroach-teamcity opened this issue Sep 8, 2021 · 0 comments · Fixed by #75845
Closed

pebble: v20.2.4: index out of range (looks like an integer overflow) #69906

cockroach-teamcity opened this issue Sep 8, 2021 · 0 comments · Fixed by #75845
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-storage Storage Team

Comments

@cockroach-teamcity
Copy link
Member

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/2633161336/?referrer=webhooks_plugin

Panic message:

panic.go:679: runtime error: index out of range [4294967264] with length 0
--
runtime.boundsError
panic.go:679: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

/usr/local/go/src/runtime/panic.go#L678-L680 in runtime.gopanic
/usr/local/go/src/runtime/panic.go#L74-L76 in runtime.goPanicIndex
https://github.com/cockroachdb/cockroach/blob/eda2309728392593162e962a61182eab6ab003ff/vendor/github.com/cockroachdb/pebble/internal/batchskl/skl.go#L295-L297 in github.com/cockroachdb/pebble/internal/batchskl.(*Skiplist).node
https://github.com/cockroachdb/cockroach/blob/eda2309728392593162e962a61182eab6ab003ff/vendor/github.com/cockroachdb/pebble/internal/batchskl/skl.go#L264-L266 in github.com/cockroachdb/pebble/internal/batchskl.(*Skiplist).newNode
https://github.com/cockroachdb/cockroach/blob/eda2309728392593162e962a61182eab6ab003ff/vendor/github.com/cockroachdb/pebble/internal/batchskl/skl.go#L232-L234 in github.com/cockroachdb/pebble/internal/batchskl.(*Skiplist).Add
https://github.com/cockroachdb/cockroach/blob/eda2309728392593162e962a61182eab6ab003ff/vendor/github.com/cockroachdb/pebble/batch.go#L527-L529 in github.com/cockroachdb/pebble.(*Batch).Delete

cockroach/pkg/storage/pebble_batch.go

Lines 239 to 241 in eda2309

p.buf = EncodeKeyToBuf(p.buf[:0], key)
return p.batch.Delete(p.buf, nil)
}
in pkg/storage.(*pebbleBatch).Clear

cockroach/pkg/kv/kvserver/replica_raft.go

Lines 1695 to 1697 in eda2309

unsafeKey := prefixBuf.RaftLogKey(idx)
if err := readWriter.Clear(storage.MakeMVCCMetadataKey(unsafeKey)); err != nil {
return false, errors.Wrapf(err, "unable to clear truncated Raft entries for %+v", newTruncatedState)
in pkg/kv/kvserver.handleTruncatedStateBelowRaft

cockroach/pkg/kv/kvserver/replica_application_state_machine.go

Lines 701 to 703 in eda2309

if res.State != nil && res.State.TruncatedState != nil {
if apply, err := handleTruncatedStateBelowRaft(
ctx, b.state.TruncatedState, res.State.TruncatedState, b.r.raftMu.stateLoader, b.batch,
in pkg/kv/kvserver.(*replicaAppBatch).runPreApplyTriggersAfterStagingWriteBatch

cockroach/pkg/kv/kvserver/replica_application_state_machine.go

Lines 483 to 485 in eda2309

// but after the write batch is staged in the batch.
if err := b.runPreApplyTriggersAfterStagingWriteBatch(ctx, cmd); err != nil {
return nil, err
in pkg/kv/kvserver.(*replicaAppBatch).Stage

cockroach/pkg/kv/kvserver/apply/cmd.go

Lines 174 to 176 in eda2309

for iter.Valid() {
checked, err := fn(iter.Cur())
if err != nil {
in pkg/kv/kvserver/apply.mapCmdIter

cockroach/pkg/kv/kvserver/apply/task.go

Lines 279 to 281 in eda2309

// Stage each command in the batch.
stagedIter, err := mapCmdIter(batchIter, batch.Stage)
if err != nil {
in pkg/kv/kvserver/apply.(*Task).applyOneBatch

cockroach/pkg/kv/kvserver/apply/task.go

Lines 246 to 248 in eda2309

for iter.Valid() {
if err := t.applyOneBatch(ctx, iter); err != nil {
// If the batch threw an error, reject all remaining commands in the
in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries

cockroach/pkg/kv/kvserver/replica_raft.go

Lines 764 to 766 in eda2309

if len(rd.CommittedEntries) > 0 {
err := appTask.ApplyCommittedEntries(ctx)
stats.applyCommittedEntriesStats = sm.moveStats()
in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked

cockroach/pkg/kv/kvserver/replica_raft.go

Lines 424 to 426 in eda2309

defer r.raftMu.Unlock()
return r.handleRaftReadyRaftMuLocked(ctx, inSnap)
}
in pkg/kv/kvserver.(*Replica).handleRaftReady

cockroach/pkg/kv/kvserver/store_raft.go

Lines 483 to 485 in eda2309

start := timeutil.Now()
stats, expl, err := r.handleRaftReady(ctx, noSnap)
removed := maybeFatalOnRaftReadyErr(ctx, expl, err)
in pkg/kv/kvserver.(*Store).processReady

cockroach/pkg/kv/kvserver/scheduler.go

Lines 225 to 227 in eda2309

if state&stateRaftReady != 0 {
s.processor.processReady(ctx, id)
}
in pkg/kv/kvserver.(*raftScheduler).worker

cockroach/pkg/kv/kvserver/scheduler.go

Lines 165 to 167 in eda2309

stopper.RunWorker(ctx, func(ctx context.Context) {
s.worker(ctx)
})
in pkg/kv/kvserver.(*raftScheduler).Start.func2

cockroach/pkg/util/stop/stopper.go

Lines 221 to 223 in eda2309

defer s.stop.Done()
f(ctx)
}()
in pkg/util/stop.(*Stopper).RunWorker.func1
/usr/local/go/src/runtime/asm_amd64.s#L1356-L1358 in runtime.goexit

/usr/local/go/src/runtime/panic.go in runtime.gopanic at line 679
/usr/local/go/src/runtime/panic.go in runtime.goPanicIndex at line 75
vendor/github.com/cockroachdb/pebble/internal/batchskl/skl.go in github.com/cockroachdb/pebble/internal/batchskl.(*Skiplist).node at line 296
vendor/github.com/cockroachdb/pebble/internal/batchskl/skl.go in github.com/cockroachdb/pebble/internal/batchskl.(*Skiplist).newNode at line 265
vendor/github.com/cockroachdb/pebble/internal/batchskl/skl.go in github.com/cockroachdb/pebble/internal/batchskl.(*Skiplist).Add at line 233
vendor/github.com/cockroachdb/pebble/batch.go in github.com/cockroachdb/pebble.(*Batch).Delete at line 528
pkg/storage/pebble_batch.go in pkg/storage.(*pebbleBatch).Clear at line 240
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.handleTruncatedStateBelowRaft at line 1696
pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaAppBatch).runPreApplyTriggersAfterStagingWriteBatch at line 702
pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaAppBatch).Stage at line 484
pkg/kv/kvserver/apply/cmd.go in pkg/kv/kvserver/apply.mapCmdIter at line 175
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).applyOneBatch at line 280
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries at line 247
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 765
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 425
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 484
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).worker at line 226
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 166
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunWorker.func1 at line 222
/usr/local/go/src/runtime/asm_amd64.s in runtime.goexit at line 1357
Tag Value
Cockroach Release v20.2.4
Cockroach SHA: eda2309
Platform linux amd64
Distribution CCL
Environment v20.2.4
Command server
Go Version ``
# of CPUs
# of Goroutines
@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Sep 8, 2021
@yuzefovich yuzefovich changed the title sentry: panic.go:679: runtime error: index out of range [4294967264] with length 0 -- runtime.boundsError panic.go:679: *withstack.withStack (top exception) pebble: v20.2.4: index out of range (looks like an integer overflow) Sep 8, 2021
@blathers-crl blathers-crl bot added the T-storage Storage Team label Sep 8, 2021
@nicktrav nicktrav mentioned this issue Sep 8, 2021
Closed
nicktrav added a commit to nicktrav/cockroach that referenced this issue Feb 2, 2022
@nicktrav
vendor: bump Pebble to 38b68e17aa97
dd277c5 
Pebble commits:

```
38b68e17 internal/batchskl: return error on index overflow
8440f290 internal/manifest: use a line sweep to optimize NewL0Sublevels
0f5acb26 sstable: add direct block reading to suffix rewriter
26856d10 db: avoid stats flake in TestMemTableReservation
b452808f sstable: Make sstable Writer.Close idempotent
17fe1a65 sstable: add RewriteKeySuffixes function
c9e6edfc db: expose metrics on count and earliest seqnum of snapshots
b958d9a7 sstable: add a writeQueue to the sstable writer
c8dad06c db: disable automatic compactions in `MetricsTest`
015f5e38 internal/rangekey: fix range key iteration bug
```

The commit `38b68e17` contains the fix for cockroachdb#69906.

Closes cockroachdb#69906.

Release note: none
This was referenced Feb 2, 2022
Merged
Merged
nicktrav added a commit to nicktrav/cockroach that referenced this issue Feb 2, 2022
@nicktrav
vendor: bump Pebble to fe22683f9088
81518da 
Relevant commits:

```
fe22683f internal/batchskl: return error on index overflow
```

This includes a fix for cockroachdb#69906.

Release note: none
@nicktrav nicktrav mentioned this issue Feb 2, 2022
Merged
nicktrav added a commit to nicktrav/cockroach that referenced this issue Feb 2, 2022
@nicktrav
vendor: bump Pebble to 38b68e17aa97
ed343ba 
Pebble commits:

```
38b68e17 internal/batchskl: return error on index overflow
8440f290 internal/manifest: use a line sweep to optimize NewL0Sublevels
0f5acb26 sstable: add direct block reading to suffix rewriter
26856d10 db: avoid stats flake in TestMemTableReservation
b452808f sstable: Make sstable Writer.Close idempotent
17fe1a65 sstable: add RewriteKeySuffixes function
c9e6edfc db: expose metrics on count and earliest seqnum of snapshots
b958d9a7 sstable: add a writeQueue to the sstable writer
c8dad06c db: disable automatic compactions in `MetricsTest`
015f5e38 internal/rangekey: fix range key iteration bug
The commit 38b68e17 contains the fix for cockroachdb#69906.
```

Closes cockroachdb#69906.

Release note: none
craig bot pushed a commit that referenced this issue Feb 2, 2022
@sumeerbhola @aayushshah15
@jeffswenson @rafiss @RichardJCai @nicktrav @stevendanna @rail @arulajmani
Merge #58261 #75746 #75766 #75804 #75822 #75843 #75845 #75857 #75865 #…
88522a3 
…75871

58261: opt,sql: use paired-joins with non-covering indexes for left joins r=rytaft a=sumeerbhola

This is done when the left outer/semi/anti join can use a
lookup join. Prior to this, when the non-covering index
could not fully evaluate the filter for left join we could
not generate a lookup join.

With this change:
- Left outer join becomes a pair of two left outer joins.
- Left semi join is a pair of inner join followed by left
  semi join.
- Left anti join is a pair of left outer join followed by
  left anti join.

Informs #55452

Release note (performance improvement): The optimizer can now
generate lookup joins in certain cases for non-covering
indexes, when performing a left outer/semi/anti join.

75746: dev: initialize submodules in `dev doctor` r=aayushshah15 a=aayushshah15

This commit adds a check to `dev doctor` to initialize submodules, like we do
in our `Makefile`.

Fixes #72247

Release note: None

75766: server: do not check decommission list for the tenant r=JeffSwenson a=JeffSwenson

Previously, the system tenant would return PermissionDenied if the
tenant's instance_id was equivalent to a decommissioned node's id.

Now, the system tenant does not check the decommissioned node list if
the incoming node_id belongs to a non-system tenant.

This PR feeds the request context down to the OnOutgoingPing and
OnIncomingPing callbacks. Previously the callbacks were using the
ambient context. The only use of the context was a storage.MVCCGet call
in nodeTombstoneStorage.IsDecommissioned.

Release note: None

75804: sql: support RESET ALL statement r=otan a=rafiss

fixes #75435

Release note (sql change): Support for the RESET ALL statement was
added. This statement resets the values of all session variables to
their default values.

75822: sql: error when setting timezone outside of postgres max utc offsets r=otan a=RichardJCai

Release note (sql change): Previously, users would be able to set
a UTC timezone offset of greater than 167 or less than -167. This
now returns an error.

Example:

SET TIME ZONE '168'
Gives error:
invalid value for parameter "timezone": "'168'": cannot find time zone "168": UTC timezone offset is out of range.

SET TIME ZONE '-168'
Gives error:
invalid value for parameter "timezone": "'-168'": cannot find time zone "-168": UTC timezone offset is out of range.

Fixes #75168

Note: If a user has already set a UTC timezone offset outside of these bounds, it will be unchanged. 

75843: c-deps/krb5: fix build for more recent versions of autoconf r=otan a=nicktrav

More recent versions of `autoconf`, when used to build `krb5`, generates
shell scripts with invalid syntax.

Fix by pulling in the [upstream patch][1] for the issue into our tree.

Closes #72529.

[1]:
krb5/krb5@f78edbe

75845: vendor: bump Pebble to 38b68e17aa97 r=jbowens a=nicktrav

Pebble commits:

```
38b68e17 internal/batchskl: return error on index overflow
8440f290 internal/manifest: use a line sweep to optimize NewL0Sublevels
0f5acb26 sstable: add direct block reading to suffix rewriter
26856d10 db: avoid stats flake in TestMemTableReservation
b452808f sstable: Make sstable Writer.Close idempotent
17fe1a65 sstable: add RewriteKeySuffixes function
c9e6edfc db: expose metrics on count and earliest seqnum of snapshots
b958d9a7 sstable: add a writeQueue to the sstable writer
c8dad06c db: disable automatic compactions in `MetricsTest`
015f5e38 internal/rangekey: fix range key iteration bug
```

The commit `38b68e17` contains the fix for #69906.

Closes #69906.

Release note: none

75857: sql: fix small race in distIndexBackfiller r=adityamaru a=stevendanna

This fixes a small race condition in
distIndexBackfiller. updateJobDetails calls SetResumeSpansInJob which
mutates the ResumeSpanList in the job details.  Normally, this is only
called from the periodic updater.  However, when the testing knob
AlwasyUpdateIndexBackfillDetails is set, we also update it on every
ProducerMetadata message we get back

Release note: None

75865: build: address util.log.logcrash package rename r=knz a=rail

After `util.log` was renamed to `util.log.logcrash`, the build system
stopped updating the Sentry environment variable properly. Instead of
setting it to the release version, it was falling back to the default
"development" value. As a result, all Sentry reports went to the
development environment bucket.

This patch addresses the name change.

Release note: None

75871: logictestccl: fix stale issue number in TODO r=arulajmani a=arulajmani

We closed #69265 in favour of #70558, and the only remaining work
left to address locality aware planning for tenants is captured
in #75864.

Release note: None

Co-authored-by: sumeerbhola <[email protected]>
Co-authored-by: Aayush Shah <[email protected]>
Co-authored-by: Jeff <[email protected]>
Co-authored-by: Rafi Shamim <[email protected]>
Co-authored-by: richardjcai <[email protected]>
Co-authored-by: Nick Travers <[email protected]>
Co-authored-by: Steven Danna <[email protected]>
Co-authored-by: Rail Aliiev <[email protected]>
Co-authored-by: arulajmani <[email protected]>
nicktrav added a commit that referenced this issue Feb 2, 2022
@nicktrav
vendor: bump Pebble to 74c112a0ac85
b6ff586 
Relevant commits:

```
74c112a0 internal/batchskl: return error on index overflow
b0a17d1d ci: enable CI for release branches
```

This includes a fix for #69906.

Release note: none
@craig craig bot closed this as completed in 13694f6 Feb 2, 2022
@craig craig bot closed this as completed in #75845 Feb 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-storage Storage Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

vendor: bump Pebble to 38b68e17aa97 nicktrav/cockroach
1 participant
@cockroach-teamcity