at startup, keep duplicates in in-memory index since they will be cleaned shortly

[jeffwashington]

Problem

See #30711

At startup, scan storages to populate index. We can easily identify pubkeys that are in multiple slots (duplicates).
Today, these duplicates are inserted into the disk index. As soon as the first clean runs, the index entries for all these items will be loaded and the older ones will be cleaned (ie. removed) and then the entries will be rewritten. This write/read/rewrite is a waste of i/o bandwidth.

Summary of Changes

Hold all items with duplicates in in-memory index since we know clean will shortly read/modify/write all of them. This improves index generation performance and the performance of the first clean.

Fixes #

[brooksprumo]

Code looks good. A few questions below.

[brooksprumo]

This comment is actually for the None arm, but I cannot comment on it (line 1068)

When inserting/updating the disk, I remember you mentioning that we don't need anything on disk with ref counts greater than 1; is that the case here with this logic (where new_ref_count is always 1 on line 1071)?

[jeffwashington]

I'm not following. The none arm is for when we are inserting pubkey A but A does not exist on disk yet. So we get a None. Then, we insert it directly to disk with a ref count of 1 (or 0 if entry is cached). This is correct and what we would expect. A second attempt at inserting will leave disk unchanged and add the duplicate to duplicates. Later, when duplicates is iterated, we will load the single entry (and 1 refcount) from disk and merge it with the contents of duplicates, increasing the refcount on the in-memory entry to 2+. The eviction caching logic will prevent us from removing the 2+ refcount entries to disk depending on the tuning of that cache.

[brooksprumo]

Basically I was trying to answer the question of "the refcount is always 1 here, right?". I didn't know/remember about the cached case, so refcount can be 0 too. Was seeing if an assert here made sense or not. I don't think anything needs to change; was endeavoring to understand more 😺

[brooksprumo]

lgtm

[codecov]

Codecov Report

Merging #30736 (c0e0fd3) into master (4285cb2) will increase coverage by 0.0%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##           master   #30736   +/-   ##
=======================================
  Coverage    81.4%    81.4%           
=======================================
  Files         723      723           
  Lines      203533   203537    +4     
=======================================
+ Hits       165845   165876   +31     
+ Misses      37688    37661   -27