Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backupccl: lower the buffer size of doneScatterCh in gen split and scatter #99867

Merged
merged 1 commit into from
Mar 30, 2023

Conversation

rhu713
Copy link
Contributor

@rhu713 rhu713 commented Mar 28, 2023

Previously, doneScatterCh in GenerativeSplitAndScatterProcessor had a large enough buffer size to never block, which was equal to the number of import spans in the restore job. This can cause restore to buffer all restore span entries in memory at the same time. Lower the limit to be numNodes * maxConcurrentRestoreWorkers, which is the max number of entries that can be processed in parallel downstream.

Release note: None

…atter

Previously, doneScatterCh in GenerativeSplitAndScatterProcessor had a
large enough buffer size to never block, which was equal to the number
of import spans in the restore job. This can cause restore to buffer all
restore span entries in memory at the same time. Lower the limit to be
numNodes * maxConcurrentRestoreWorkers, which is the max number of
entries that can be processed in parallel downstream.

Release note: None
@rhu713 rhu713 requested a review from a team as a code owner March 28, 2023 21:33
@rhu713 rhu713 requested review from benbardin and removed request for a team March 28, 2023 21:33
@blathers-crl
Copy link

blathers-crl bot commented Mar 28, 2023

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@rhu713 rhu713 requested review from dt and lidorcarmel and removed request for benbardin and dt March 28, 2023 21:33
@rhu713 rhu713 added the backport-23.1.x Flags PRs that need to be backported to 23.1 label Mar 30, 2023
@rhu713
Copy link
Contributor Author

rhu713 commented Mar 30, 2023

bors r+

@craig
Copy link
Contributor

craig bot commented Mar 30, 2023

Build succeeded:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-23.1.x Flags PRs that need to be backported to 23.1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants