ShrinkCandidates only holds slot #33173
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jeffwashington
force-pushed
the
sp5
branch
2 times, most recently
from
cc8b30c to
c4e5848
Compare
Codecov Report
@@ Coverage Diff @@
## master #33173 +/- ##
=========================================
- Coverage 82.1% 82.1% -0.1%
=========================================
Files 785 785
Lines 211205 211194 -11
=========================================
- Hits 173473 173420 -53
- Misses 37732 37774 +42 |
|
fyi, had to rebase against the other pr that just went in. The goal of the other one was to reduce the # line changes in this one and get rid of noisy useless changes. |
brooksprumo
approved these changes
Sep 7, 2023
Comment on lines
+4295
to
+4297
| let Some(store) = self.storage.get_slot_storage_entry(*slot) else { | ||
| continue; | ||
| }; |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Improving startup time.
Now that we only have 1 storage per slot, it is more efficient to not hold an
Arcto the append vec in the shrink list.During index generation, especially with incremental snapshots, we can end up with say 60M duplicate pubkeys across all 432k slots. This results in a lot of record keeping and inefficient
HashMapstuffing of almost every(Slot, Arc<AppendVec>). Getting the append vecs is not fast. They can be looked up in the bg thread by shrink itself. A slot is sufficient to request a shrink.Summary of Changes
Just hold a
HashSet<Slot>instead of aHashMap<Slot, Arc<AppendVec>>Fixes #