Perf/HalfPath state db key

[asdacap]

Prefix state db entry with part of path

• Includes Perf/Live Pruning #6439, Allow configuring last state to be kept #6809, Fix/long running iterator #6840
• Inspired by one of the guy I interviewed recently, what if we keep the whole hash db/pruning, but improve its data locality by ordering the keys similar to flat db layout.
• Well, you could, by prefixing the hash with part of the path (you don't need all the path).
• This is the default for fresh state db. If old hash key is needed, --Init.StateDbKeyScheme Hash can be specified. An explicit --Init.StateDbKeyScheme HalfPath is needed for full pruning to migrate to the new key layout.
• Key structure look like this. For state (total 42 byte)

+--------------+------------------+------------------+--------------+
| section byte | 8 byte from path | path length byte | 32 byte hash |
+--------------+------------------+------------------+--------------+

• For storage (total 74 byte)

+--------------+---------------------+------------------+------------------+--------------+
| section byte | 32 byte from address | 8 byte from path | path length byte | 32 byte hash |
+--------------+---------------------+------------------+------------------+--------------+

• The section byte is:
  • 0 if state and path length is <= 5.
  • 1 if state and path length is > 5.
  • 2 if storage.

The keys are separated with section byte due to the different characteristics of these nodes. The idea being that top level
node can be up to 5 times bigger than lower node, and grew a lot due to pruning. So mixing them makes lower
node sparser and have poorer cache hit, and make traversing leaves for snap serving slower.

Performance

• As it now need to keep track of the path, some extra compute and allocation is needed.
• However, the improved IO efficiency reduces the overall block processing time by up to 50%.
• In memory pruning now need to keep track of the path. This add extra memory use and cause previously deduplicated node by hash, to no longer get deduplicated. This seems to add about 25% per-node memory usage.
• Because of this, when it detect that current state is using hash keys, and HalfPath layout is not explicitly mentioned, in memory pruning will only keep track of the hash causing it to behave as before.
• In practice, it does not really make much of a difference in block processing time.
• Graph is (master, after with existing hash db, after with --Init.StateDbKeyScheme HalfPath to also track path, after with HalfPath)x4. The last 2 set have 8GB memory hint.
  
• If you squint really hard, the CPU time seems to reduce and flatten a little bit.

Sync performance

• Reduced compaction. Bytes write per node reduced from arount 1KB to 0.5KB.
• Reduce disk space from 196G to 168G,
• Sync time tend to reduce a little but inconsistent.
• Memory allocation per node increased by 25%.
• Graph is master, (after but with hash), after
  

Migration.

• Existing user can run with --Init.StateDbKeyScheme HalfPath and then run full pruning to migrate.
• This is not enabled by default as in memory pruning will need to already keep track of the path at start. That said, there aren't that much of an overhead (in term of block processing) if this is enabled by default.
• This also cause the database copied to track path which significantly increase memory requirement for batched trie visitor's job pool. Without increasing the memory budget, iops requirement increase significantly. Because of this, migration via full pruning is not recommended.

Database growth

• Note: See Perf/Live Pruning #6439
• Because of the need to track path, on the same memory budget, it will persist more nodes.
• However, it seems that due to improved compression ratio, the size of the databse grows slower.
• In this test, starting block is 18310000 from 9 Oct, and it was stopped on block 18400000 (21 Oct).`

	Persisted Nodes (Million)	Size before (GB)	Size after (GB)	Increase (GB)	Per node increase (byte)	Increase (%)
Hash	193	173.3	205.6	32.3	167.4	18.64%
HalfPath	271	143.7	157.8	14.1	52.0	9.81%

• When in memory pruning is disable, similar reducing in database increase is also seen:

	Persisted Nodes (Million)	Size before (GB)	Size after (GB)	Increase (GB)	Per node increase (byte)	Increase (%)
Hash	1270	173.3	284.9	111.6	87.9	64.40%
HalfPath	1240	143.7	199.9	56.2	45.3	39.11%

Full pruning

• Uses far less IO and memory.
• Unthrottled troughput remains about the same.
• Default setting (throttled) throughput increased by about 50%.
• Graph is before, after, before, after
  

Changes

• Includes Fix double write during full pruning #6415 (needed for full pruning to migrate key)
• Includes Replace caching store with row cache #6405 (make code slightly simpler)
• Created a class TreePath for representing path. A ref TreePath parameters is added everywhere in PatriciaTree and TrieNode. On traversal to child, the tree path is mutated to add necessary key to become the correct path for its child and it is mutated back to its parent after returning from traversing its child. Its messy, but it significantly reduce memory allocation overhead.
• For the storage, the PatriciaTree and friends, accept a IScopedTreeStore which is a subinterface of INodeResolver. The implementation, ScopedTreeStore keep the hash of the storage, which is then proxied to ITreeStore that we've known and love.
• ITreeStore has been modified to also accept the storage and path in addition to the standard hash.
• TreeStore has been modified to be keyed by Hash256?, TreePath, Keccak tuple instead of just Keccak.
• An abstraction INodeStorage replaces IKeyValueStore as the storage layer of the nodes, accepting the storage root and the path also. This is the class the determine the key and allow fallback read as well as configuring between hash and halfpath.
• Sync code has been modified to interact with INodeStorage instead of IKeyValueStore.

Types of changes

What types of changes does your code introduce?

• Optimization
• Refactoring

Testing

Requires testing

• Yes
• No

If yes, did you write tests?

• Yes
• No

Notes on testing

• Seems to work.

Documentation

Requires documentation update

• Maybe?

Requires explanation in Release Notes

• Yes

New state db key structure improve block processing time by up to 50%. Existing users are unaffected.

[kamilchodola]

@asdacap Does this change requires resync of the DB?

[asdacap]

It require resync to take effect. But we can add a fallback read, so it can be made to work without resync. However, we are effectively doing two read at each node which is not great. I guess the first read would get bloom filtered, but still not great.

[asdacap]

Complication was removed. State item always need path now.

[benaadams]

Have worked with code base and made changes, so would put more weight on other reviewers

[LukaszRozmej]

Seeing a lot of Hash256, would it be better to move to ValueHash256 in those places?

Most my comments are just nitpicks.

[asdacap]

So, two place where Hash256 is kept:

• TrieNode's Keccak. -> Not gonna touch that here.
• The storage address -> Storage address is likely shared between multiple nodes so it make sense to use a reference type so that each node only use 8 byte.

[Scooletz]

A few initial comments, for better understanding of the decisions taken here.

[Scooletz]

I'd move this code elsewhere. It's dependent on the TrieStore.Schemeso maybe TrieStore could provideFullDbScanFlags`?

[asdacap]

Fair point. NodeStorage already change the readahead flag already.

[Scooletz]

One minor and some comments.