Adds calculate_incremental_accounts_hash() #29734
Conversation
brooksprumo
force-pushed
the
iah/calculate-iah
branch
from
cbfe4ea to
267ee01
Compare
|
how many slots were in the incremental hash calculation in your benchmark? # of slots can range from 1-25k I think. |
| @@ -2187,7 +2188,7 @@ impl<'a> AppendVecScan for ScanState<'a> { | |||
| } | |||
|
|
|||
| impl AccountsDb { | |||
| pub const ACCOUNTS_HASH_CACHE_DIR: &str = "calculate_accounts_hash_cache"; | |||
| pub const ACCOUNTS_HASH_CACHE_DIR: &str = "accounts_hash_cache"; | |||
There was a problem hiding this comment.
maybe we need to delete 'calculate_accounts_hash_cache' on user's ledger?
There was a problem hiding this comment.
Yeah, I was wondering how to handle this as well. I should've called it out here.
Since we'll need separate caches for "full" and "incremental" account hashes, which already breaks/changes the current path, I figured shortening this to "accounts_hash_cache" would be nice.
Want me to add code to check for "calculate_accounts_hash_cache" and delete the directory if it's found?
There was a problem hiding this comment.
not sure. just seems like a gotcha. this folder could be 30G today on mnb?
Not sure, I'll need to re-run the validator to get exact numbers. I ran it long enough for my node to produce multiple full snapshots, so the whole range of 1-25k should've been exercised. The 3 seconds number that I quoted was from the times that I check my logs, but I'm not sure the number of storages at each of those datapoints. The metrics should match the number of slots though. Want me to get specific numbers? Edit: We talked offline and looked over metrics in the logs. |
Problem
It is not possible to calculate an incremental accounts hash.
Summary of Changes
calculate_incremental_accounts_hash()Numbers
I ran a modified validator on my dev box that calculated the incremental accounts hash in AHV after calculating the full accounts hash and the averages times were:
While likely not exact benchmark numbers, it was at least good to see significant speedup with IAH.
Node Operators
Node operators, please delete your
<ledger>/calculate_accounts_hash_cachedirectory (if you have one). It has been moved/renamed to<ledger>/accounts_hash_cache/full.