-
Notifications
You must be signed in to change notification settings - Fork 622
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ReshardingV3] State Witness - new state root generation #12074
Comments
This was referenced Sep 11, 2024
github-merge-queue bot
pushed a commit
that referenced
this issue
Sep 20, 2024
Create the code flow intended for resharding state root creation. The ultimate goal will be the temporary memtries created, which will be used until new memtries are created from flat storage. This will be a multi-step work. To complete it, all mentioned TODOs must be resolved. ### Overview The core is a `MemTrieUpdate::retain_multi_range` function and a simple `test_retain_multi_range` test. It follows the logic in #12074. I plan to enhance logic sequentially until it will be fit for the actual resharding. Drawback is that code is not used on chain, but setting up full resharding test requires separate work. I suggest to start from the the incomplete flow; hopefully I will need ~5 PRs to set up the full test. It follows the "top-down" logic as in insert/delete functions: first we pull root node to in-memory update list, then we descend into children. In the end, we call `to_trie_changes` which calls `post_order_traverse_updated_nodes` to create full set of changes to be applied to trie. ### Alternative For the multi-range deletion, better logic is "bottom-up": we first understand the result of children subtrees, and then mark root node as updated if needed. It's not clear whether everything must be "bottom-up". It probably has its own challenges. But here it is strictly better because it automatically gives the post traversal order of updated nodes. We could compute hashes on the fly but it's less LoC to postpone it to reuse existing logic, so I just call `compute_hashes_and_serialized_nodes` in the end.
github-merge-queue bot
pushed a commit
that referenced
this issue
Sep 24, 2024
Next step on #12074. Supporting all cases I came up with where nodes restructuring is required. Surprisingly, `squash_node` is enough to call. I only needed to implement trivial cases which weren't possible for single key deletion. I implemented the tests first, and majority of them failed before changing the logic. Each test is comparing naive approach with `retain_multi_range`. ### Notes * I'm a bit scared that I didn't realise the need to squash Extension before. Fortunately, `squash_node` handles that, but if you feel some cases are not covered here, feel free to post suggestions! * Reused + copypasted some Robin' tooling to generate interesting nodes conversions. * Note that test for reading "extra" child node is not required because we always read all children. ### Next steps * A bit more testing * Similar logic for partial trie * Generating intervals needed for resharding * Use that to implement shard switch on chain
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We agreed that the nicest approach is to have the same code handling both resharding proof generation and verifying, similarly to trie read-write operations.
For now we describe only memtrie logic. Partial trie logic is TODO.
We will implement function against trie storage:
GenericTrie::cut(state_root, boundary_account, retain: Left/Right) -> (new_state_root, proof)
which main logic will be
GenericTrie::delete_multi_range(state_root, intervals: { [left_key, right_key) }) -> (new_state_root, proof)
cut
Simulates cutting out all trie keys by
boundary_account
, retaining left or right part based onretain
parameter. To process resharding, if node tracks the left child, it needs to call this function forretain = Left
to get new state root with proof, and vice versa for the right child.Because trie keys start from different trie type bytes, we must execute several range removals.
For each trie key type
t
which includes account id (except PromiseYield), we remove the interval:[ [t] + boundary_account, [t + 1] )
[ [t], [t] + boundary_account )
If trie key doesn't have account id, we do nothing. This is because we fully copy delayed, buffered and promise-yield receipts and discard them later.
We supply the resulting list of ranges for removal to the function below.
delete_multi_range
Recursively descends into children, removing all keys falling into
intervals
. Returns new state root and proof of deletion.TODO: because the trie access logic is complex, I can't describe it using existing structs. For now, I will use only memtrie. But probably
MemTrieUpdate
can be reused.delete_multi_range
callsdelete_multi_range_recursive(state_root: MemTrieNodeId, current_key: Vec<u8>, intervals, accesses: &mut TrieAccesses)
. This method will probably returnMemTrieChanges
. This itself will be given toapply_memtrie_changes
to generate new state root. Initiallycurrent_key = vec![]
,accesses
are empty.TODO: not sure if
MemTrieChanges
fits here well.delete_multi_range_recursive
:We need to make decision what to do with current range of keys. First, we decide separately on each interval
[L, R)
against currentkey
. There are 3 decisions we can make.Note that subtree of
key
can contain only strings resulting by appending new symbols tokey
.[L, R)
. Example: key = 92, L = 92c. key < L, but its subtree may contain 92c, 92c0, etc - strings which fall into[L, R)
.The final decision is as follows:
In the case of Descend, we visit all children, depending on the node type, collect resulting nodes and compute new node (possibly compress Branch to Extension).
For each node read we put it to
TrieAccesses
. These nodes form a proof. (Check: we don't need to read values, right?)Note: looks like proof is essentially all paths to the intervals' boundaries. But I don't see a simple way to define
delete_multi_range
in terms of existing trie queries.Next steps
Out of scope
Deduplicate memtrie & partial trie logic - for example,
memtrie_lookup
andlookup_from_state_column
. It could be possible to unite these impls using templates. But for now memtrie is actively worked on by CR Team and after applied optimisations it may be hard to unite logic.The text was updated successfully, but these errors were encountered: