crl-release-23.1: db: improve delete pacing #2696
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Delete pacing is currently an on or off decision. If we are running out of space or have too many obsolete bytes in relation to live bytes, we disable pacing. Otherwise we pace at the configured rate (128MB by default in CRDB). This change improves pacing by keeping track of the average deletion rate over the last 5 minutes and increasing the target rate to match this rate if necessary. The intention is to avoid deletions lagging behind. Informs cockroachdb#2662.
|
This change is |
|
Note: this is a backport of #2673, adapted to the 23.1 codebase. |
jbowens
approved these changes
Jun 30, 2023
There was a problem hiding this comment.
Reviewed 5 of 5 files at r1, all commit messages.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on @RaduBerinde)
RaduBerinde
added a commit
to RaduBerinde/cockroach
that referenced
this pull request
Jun 30, 2023
…598cb059 c0a2ede2 db: improve delete pacing (cockroachdb/pebble#2696) Informs cockroachdb#2662. Release justification: fixes very high goroutine count on heavy write workloads with fast disks. Release note (bug fix): On heavy write workloads with fast disks, the deletion of obsolete LSM files could fall behind, causing high goroutine count. Pacing of obsolete file deletion now adapts based on recent deletion rate to avoid falling behind.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Delete pacing is currently an on or off decision. If we are running out of space or have too many obsolete bytes in relation to live bytes, we disable pacing. Otherwise we pace at the configured rate (128MB by default in CRDB).
This change improves pacing by keeping track of the average deletion rate over the last 5 minutes and increasing the target rate to match this rate if necessary. The intention is to avoid deletions lagging behind.
Informs #2662.
I have tested this change against a heavy write workload where deletion is known to lag behind (resulting in 10k+ goroutines waiting in the pacer); with the change, the goroutine count is small and stable (after the first 5-10 minutes):
