Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pebble: fatal commit error: allocation failed because arena is full #102945

Closed
cockroach-teamcity opened this issue May 9, 2023 · 4 comments
Closed

pebble: fatal commit error: allocation failed because arena is full #102945

cockroach-teamcity opened this issue May 9, 2023 · 4 comments
Assignees
@sumeerbhola
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-storage Storage Team
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 9, 2023 •
edited by cockroach-jira-scripts
Loading

kv/kvserver.TestStoreRangeUpReplicate failed with artifacts on release-23.1 @ 5376e479204d6e8243f67c30aea3d031df529afd:

F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x61a
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !goroutine 397565 [select]:
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Start.func2({0xc00102a400, 0x49e093})
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:1997 +0x11a
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x1f7
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x61a
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !goroutine 395252 [select]:
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc0047b3760, {0xcdbd610, 0xc00501cff0?})
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:498 +0x4bd
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x1f7
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x61a
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !goroutine 395009 [select]:
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0xcdbd610, 0xc004185aa0})
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:235 +0x14e
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x1f7
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x61a
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !For more context, check log files in: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestStoreRangeUpReplicate272463188
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !For more context, check log files in: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestStoreRangeUpReplicate272463188
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !****************************************************************************
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !This node experienced a fatal error (printed above), and as a result the
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !process is terminating.
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !problem in CockroachDB. With your help, the support team at Cockroach Labs
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !will try to determine the root cause, recommend next steps, and we can
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !improve CockroachDB based on your report.
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !Please submit a crash report by following the instructions here:
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !    https://github.com/cockroachdb/cockroach/issues/new/choose
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !If you would rather not post publicly, please contact us directly at:
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !    [email protected]
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !
F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1 !The Cockroach Labs team appreciates your feedback.

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/replication

This test on roachdash | Improve this report!

Jira issue: CRDB-27750
@cockroach-teamcity cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv-replication labels May 9, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone May 9, 2023
@erikgrinaker
Copy link
Contributor

erikgrinaker commented May 9, 2023 •
edited
Loading

This one's new:

F230509 12:42:26.861341 397509 3@pebble/db.go:811  [n2,s2,pebble] 1  pebble: fatal commit error: allocation failed because arena is full

@cockroachdb/storage Do you know what's up with this? Didn't flush to disk fast enough?

This was referenced May 9, 2023
Closed
Closed
Closed
Closed
Closed
Closed
@mgartner mgartner changed the title kv/kvserver: TestStoreRangeUpReplicate failed pebble: fatal commit error: allocation failed because arena is full May 9, 2023
This was referenced May 9, 2023
Closed
Closed
@cucaroach cucaroach mentioned this issue May 9, 2023
Closed
@blathers-crl blathers-crl bot added the T-storage Storage Team label May 9, 2023
@jbowens jbowens mentioned this issue May 9, 2023
Closed
@cucaroach cucaroach mentioned this issue May 9, 2023
Closed
@jbowens jbowens added the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label May 9, 2023
@cucaroach cucaroach mentioned this issue May 9, 2023
Closed
This was referenced May 9, 2023
Closed
Closed
Closed
Closed
Closed
Closed
@sumeerbhola
Copy link
Collaborator

Investigation ongoing -- see https://cockroachlabs.slack.com/archives/CAC6K3SLU/p1683644375098049

@tbg tbg added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label May 10, 2023
@tbg
Copy link
Member

tbg commented May 10, 2023

Will be fixed by cockroachdb/pebble#2517

@stevendanna stevendanna mentioned this issue May 10, 2023
Closed
@tbg tbg mentioned this issue May 10, 2023
Closed
This was referenced May 10, 2023
Closed
Closed
sumeerbhola added a commit to cockroachdb/pebble that referenced this issue May 10, 2023
@sumeerbhola
db: intialize memTable.reserve when creating memTable
fe7ae7a 
Informs cockroachdb/cockroach#102945

Fixes #2516
sumeerbhola added a commit to sumeerbhola/pebble that referenced this issue May 10, 2023
@sumeerbhola
db: intialize memTable.reserve when creating memTable
e3c187c 
Informs cockroachdb/cockroach#102945

Fixes cockroachdb#2516
@sumeerbhola sumeerbhola mentioned this issue May 10, 2023
Merged
sumeerbhola added a commit to cockroachdb/pebble that referenced this issue May 10, 2023
@sumeerbhola
db: intialize memTable.reserve when creating memTable
23f478b 
Informs cockroachdb/cockroach#102945

Fixes #2516
jbowens added a commit to jbowens/cockroach that referenced this issue May 10, 2023
@jbowens
go.mod: bump Pebble to 23f478ba445a
95869f7 
```
23f478ba db: intialize memTable.reserve when creating memTable
```

Fixes: cockroachdb#102945
Release note: none
Epic: none
@jbowens jbowens mentioned this issue May 10, 2023
Merged
jbowens pushed a commit to jbowens/pebble that referenced this issue May 10, 2023
@sumeerbhola @jbowens
db: intialize memTable.reserve when creating memTable
ee4fc54 
Informs cockroachdb/cockroach#102945

Fixes cockroachdb#2516
This was referenced May 10, 2023
Merged
Merged
jbowens added a commit to jbowens/cockroach that referenced this issue May 10, 2023
@jbowens
go.mod: bump Pebble to fe7ae7a62e0f
d0e3690 
```
fe7ae7a6 db: intialize memTable.reserve when creating memTable
7a72be04 objstorage: remove leading 0s from creator ID
049725f4 objstorage: prepend a hash value to object names
8388e22a vfs: unify linux implementation across arm and !arm
```

Informs: cockroachdb#102945
Release note: none
Epic: none
jbowens pushed a commit to cockroachdb/pebble that referenced this issue May 10, 2023
@sumeerbhola @jbowens
db: intialize memTable.reserve when creating memTable
0e62143 
Informs cockroachdb/cockroach#102945

Fixes #2516
This was referenced May 10, 2023
Closed
Closed
craig bot pushed a commit that referenced this issue May 10, 2023
@ecwall @jbowens
Merge #102982 #103025
230a61b 
102982: roachtest: ignore TestContextCancelExec r=rafiss a=ecwall

Fixes #102674

Ignore TestContextCancelExec which has a race between context cancellation and query execution here:
https://github.com/lib/pq/blob/381d253611d666974d43dfa634d29fe16ea9e293/go18_test.go#L92

Release note: None

103025: go.mod: bump Pebble to fe7ae7a62e0f r=jbowens a=jbowens

```
fe7ae7a6 db: intialize memTable.reserve when creating memTable
7a72be04 objstorage: remove leading 0s from creator ID
049725f4 objstorage: prepend a hash value to object names
8388e22a vfs: unify linux implementation across arm and !arm
```

Informs: #102945
Release note: none
Epic: none

Co-authored-by: Evan Wall <[email protected]>
Co-authored-by: Jackson Owens <[email protected]>
This was referenced May 10, 2023
Closed
Closed
@jbowens
Copy link
Collaborator

jbowens commented May 10, 2023

Fix has been merged to master and crl-release-23.1

@jbowens jbowens closed this as completed May 10, 2023
@mgartner mgartner mentioned this issue May 10, 2023
Closed
@tbg tbg mentioned this issue May 11, 2023
Closed
@jbowens jbowens moved this to Done in [Deprecated] Storage Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sumeerbhola sumeerbhola

Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-storage Storage Team
Projects
No open projects
[Deprecated] Storage
Archived in project
Milestone
23.1
Development

No branches or pull requests
5 participants
@erikgrinaker @jbowens @tbg @cockroach-teamcity @sumeerbhola