-
Notifications
You must be signed in to change notification settings - Fork 2.6k
Further storage iterator refactoring #13445
Further storage iterator refactoring #13445
Conversation
…ey_values_while`
…child_keys_with_prefix`
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
primitives/state-machine/src/lib.rs
Outdated
.apply_to_key_values_while( | ||
child_info, | ||
let iter = proving_backend | ||
.keys(IterArgs { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
.keys(IterArgs { | |
.pairs(IterArgs { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These values aren't actually used in the loop though, so isn't this just a waste to also fetch the values?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should change the size of the proof.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm.... okay, then we should have a test for this. I'll add one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cheme I might be misunderstanding something here, but I've added extra asserts which check the generated proof sizes (with values generated before this PR and my previous PR) and after applying this PR the test still passes, so it looks like the size of the proof remains unchanged? What am I missing? Am I doing something wrong here? Any hints on how I can test this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The trie you iterate on need to be V1 and values in trie needs to be of size > 32 bytes (so key iterator only include their hashes in the proof).
(let state_version = StateVersion::V0; -> let state_version = StateVersion::V1; in your test)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
actually the method really need to be iterating on key and values, it is important that values are included in the proof (used to sync state in after a warp sync).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! I'll try to do that.
actually the method really need to be iterating on key and values
Yeah, I'm not really doubting the necessity of doing this. I just really don't want to make this change blindly without leaving a test behind. The next person changing this code could make the same mistake and this is a really subtle trap and currently (as evidenced by the green CI) not tested at all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, I've fixed the issue, added a comment, and added a test.
I've ran the test before all my changes (both PRs) and it passes. After this PR without the fix it fails. With the fix it passes. So now this is actually tested.
@cheme Thanks for your help!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you're very welcome (this is hard to guess and a bit undertested).
Downstream showed up that #13284 incurs an additional |
Thanks for letting me know; I'll verify this. |
@ggwpez This is where the extra reads are coming from:
If I'm reading this right then the |
bot merge |
* Remove `Backend::apply_to_key_values_while` * Add `IterArgs::start_at_exclusive` * Use `start_at_exclusive` in functions which used `Backend::apply_to_key_values_while` * Remove `Backend::apply_to_keys_while` * Remove `for_keys_with_prefix`, `for_key_values_with_prefix` and `for_child_keys_with_prefix` * Remove unnecessary `to_vec` calls * Fix unused method warning in no_std * Remove unnecessary import * Also check proof sizes in the test * Iterate over both keys and values in `prove_range_read_with_size` and add a test
Awesome work @koute (cc @bkchr ), I've done a basic benchmark (not accurate as it was hard to control the cache):
The paritydb improvement is impressive :) |
* Remove `Backend::apply_to_key_values_while` * Add `IterArgs::start_at_exclusive` * Use `start_at_exclusive` in functions which used `Backend::apply_to_key_values_while` * Remove `Backend::apply_to_keys_while` * Remove `for_keys_with_prefix`, `for_key_values_with_prefix` and `for_child_keys_with_prefix` * Remove unnecessary `to_vec` calls * Fix unused method warning in no_std * Remove unnecessary import * Also check proof sizes in the test * Iterate over both keys and values in `prove_range_read_with_size` and add a test
Followup of #13284
This PR further cleans up the storage backend trait and gets rid of (now unnecessary) callback-based iteration methods.