Perf/Live Pruning

[asdacap]

• We have seen Perf/HalfPath state db key #6331 reduces database growth by about 50% likely due to improved compression. Branch that change only one child that sits next to previously unchanged branch probably get compressed really well.

• BUT WAIT THERE'S MORE!

• Since different path no longer share hashes we can approximately keep track of what is currently in the database, and on persist of a canonical block, we remove the old value.

• This work suprisingly well, where 90% (95% for state, 82% for storage) of the about to be persisted key have a match in the LRU for tracking persisted node.

• This does require additional memory with an LRU of 2 million nodes. This should take about 2Mil * 48 byte of memory at best case, with an additional 0.5Mil * 48 byte to keep track of recommitted nodes to prevent them from being deleted. Only path of nibble length <= 14 is tracked to save space.

• Note, this does mean that for storage it will need to keep the full 32 byte address for the key, to prevent two storage from accidentally sharing the same (key, hash) tuple, which is unlikely but a vector for attack. This increase disk space use by about 2GB. Block processing time is much harder to notice any difference, but there could be other overhead that is not obvious right now.

• In a test run, running blocks from Oct 9 to Oct 21 the database does grow slower but only by half.

	Size before (GB)	Size after (GB)	Increase (GB)	Increase (%)
HalfPath	143.7	157.7	14	9.74%
HalfPath with improved pruning	143.7	151.4	7.7	5.36%

• However, looking deeper and scanning the database shows that the uncompressed database size only grow by 1.28% instead of 14.62% under normal case. This is likely due to how rocksdb works where deletion is not performed until the SST file is merged/compacted with lower level SST file.

	Uncompressed Size GB (Before)	Uncompressed Size GB	Increase (GB)	Increase
HalfPath	217	253.780	37.095	14.62%
HalfPath with improved pruning	217	219.499	2.814	1.28%

• On a longer run, with block spanning from 9 Oct to about 20 Jan (about 3 month), state db size grew from 143GB to 164GB.

• You can also see that the delete command saturated enough of the database after about 2 week.
  

• Halfpath without node removal grew from 143GB to 262GB.
  

• On master (hash layout), state db grew from 174GB to 428GB.
  

Changes

• List the changes

Types of changes

What types of changes does your code introduce?

• Bugfix (a non-breaking change that fixes an issue)
• New feature (a non-breaking change that adds functionality)
• Breaking change (a change that causes existing functionality not to work as expected)
• Optimization
• Refactoring
• Documentation update
• Build-related changes
• Other: Description

Testing

Requires testing

• Yes
• No

If yes, did you write tests?

• Yes
• No

Notes on testing

Optional. Remove if not applicable.

Documentation

Requires documentation update

• Yes
• No

If yes, link the PR to the docs update or the issue with the details labeled docs. Remove if not applicable.

Requires explanation in Release Notes

• Yes
• No

If yes, fill in the details here. Remove if not applicable.

Remarks

Optional. Remove if not applicable.