Hash stored accounts in bg

[jeffwashington]

Problem
While processing a slot, the same account can be written to many times.
Hashing accounts is not free and gets expensive with large data. Hashing an account that is not the final update is not usually necessary.

Summary of Changes
When an account is stored to the accounts_cache, don't hash the contents first - instead use None. When a caller requires a hash, and a hash hasn't been calculated, calculate it and store it. A background thread processes items that are added to the accounts_cache, calculates the hash and updates the cached item in accounts_cache.

Fixes #

[jeffwashington]

this pr: (improvement is in store_us and overall time)
ledger processing timing ExecuteTimings load_us 10909296 execute_us 55929281 store_us 5,725,080 details ExecuteDetailsTimings serialize_us 10248310 create_vm_us 1681543 execute_us 7627294 deserialize_us 13642778 changed_account_count 57938 total_account_count 1113864 total_data_size 48123505204 data_size_changed 49975074
ledger processed in 34 seconds. 4571 MB allocated. root slot is 69585630 1 fork at 69585662 with 32 frozen banks

master:
ledger processing timing ExecuteTimings load_us 10516422 execute_us 54218813 store_us 21,934,006 details ExecuteDetailsTimings serialize_us 9357363 create_vm_us 1677496 execute_us 7424016 deserialize_us 12894874 changed_account_count 57938 total_account_count 1113864 total_data_size 48123505204 data_size_changed 49975074
ledger processed in 41 seconds. 5092 MB allocated. root slot is 69585630 1 fork at 69585662 with 32 frozen banks

[jeffwashington]

first prototype: #15617

[codecov]

Codecov Report

Merging #16157 (829231e) into master (527adbe) will increase coverage by 0.0%.
The diff coverage is 85.7%.

@@           Coverage Diff           @@
##           master   #16157   +/-   ##
=======================================
  Coverage    80.0%    80.0%           
=======================================
  Files         410      410           
  Lines      109506   109592   +86     
=======================================
+ Hits        87674    87761   +87     
+ Misses      21832    21831    -1     

[carllin]

Ah clever, so if the hash isn't computed yet by the background service by this point, (for things like bank.freeze() that need the hash), it'll compute it here.

[jeffwashington]

Thanks. I am just implementing the required blocking, lazy behavior in a way that seems correct, consistent, performant. I think it fits Rust idioms sufficiently.

[sakridge]

looks good and super cool to have higher performance

[mergify]

automerge label removed due to a CI failure