Rework storage iterators #13284

koute · 2023-01-31T14:17:51Z

This PR reworks the way we iterate over storage.

Summary of the changes:

Improved performance. Iterating over storage is now twice as fast.
Removed dangerous methods which internally iterate over the storage and collect the data into a Vec. There's no real good reason to have those (except maybe in tests), and unless extremely careful using them just results in things exploding.
Code is now simplified. The sp_state_machine::Backend trait now has methods which return proper iterators. All of the iteration methods in there (apply_to_key_values_while, for_keys_with_prefix, pairs, keys, child_pairs, etc.) are now default implemented on the trait itself, and are channeled through the raw_iter method. This both is simpler to use (you can just use an iterator now, instead of having to use one of the methods which take a callback) and simpler to implement (you don't have to reimplement the whole zoo of methods which all do the same thing - iterate over the storage - but in subtly different ways).
Fixed a bug in the benchmarking code where iteration wasn't accounted for when counting database reads. (cc @ggwpez)

This requires a new version of trie-db (paritytech/trie#181).

Potential future work

Completely remove the for_keys_with_prefix, etc. Since the trait now exposes proper iterators those are now completely unnecessary. I would have done this in this PR, but the diff's already ~1.5k lines long, so I didn't want to bloat it up any further. I'll do it in the next PR once this one is merged.
Further improve the performance when iterating. (In particular, optimize the host functions using a trick suggested by @cheme by keeping the raw iterator alive between calls and resuming it if the key matches.)

polkadot companion: paritytech/polkadot#6653

…ge_iterators

melekes

lgtm

client/db/src/bench.rs

primitives/state-machine/src/trie_backend.rs

koute · 2023-02-03T14:15:26Z

client/api/src/backend.rs

-	prefix: Option<StorageKey>,
-	current_key: Vec<u8>,
-	_phantom: PhantomData<Block>,
+	skip_if_first: Option<StorageKey>,


Note: the iterators on the storage backend and on Client work differently regarding the start_at.

For the backend if a start_at is specified then it will seek to that key and it will always include that key in the iterator's output. For the Client if the key specified in start_at matches exactly the first key it encounters then it skips that key.

This inconsistency is a little confusing, but that's how it works.

I've added some extra tests that verify this behavior, and I've also checked that they pass before my changes.

Yes that's a point to be extremely careful with, retaining the sometime ~ behavior of iteration calls from the existing host functions (all that is touched by clear_prefix from sp_externalities).
I remember something ~ with the way error are handled, but I may be confusing with something else.

Yeah, I'll probably run a full burn-in before merging this just to make sure nothing's broken.

…ge_iterators

melekes

👍

client/api/src/backend.rs

primitives/state-machine/src/trie_backend_essence.rs

cheme · 2023-02-06T17:02:42Z

primitives/state-machine/src/trie_backend_essence.rs

+	}
+
+	fn was_complete(&self) -> bool {
+		matches!(self.state, IterState::FinishedComplete)


what about FinishedIncomplete?

If it finished incomplete it didn't finish completely :D

cheme · 2023-02-06T17:12:56Z

primitives/state-machine/src/trie_backend_essence.rs

-		};
-		if let Err(e) = result {
-			debug!(target: "trie", "Error while iterating by prefix: {}", e);
+		if self.root == Default::default() {


It's always a bit awkward to tell if self.root is vec![0; 32] or hash(trie_empty_node) (eg MemoryDb::null_node_data), actually it looks like self.empty could be use instead (that is the root of a empty trie).

Hmmm... why do we have self.empty anyway? In practice won't H::default always point to an empty storage root and a value which doesn't exist? (And idiomatically T::default() is essentially always used to denote an empty something, e.g. Vec::default, HashMap::default, etc. are all empty, so it's kinda weird to have two things which mean "empty".)

But anyhow, as far as I can see changing this to self.root == self.empty will fail at least one test (pairs_are_empty_on_empty_storage) because the root there is initialized with Default::default, so the fake empty key self.empty won't match, so it will go and call TrieDBRawIterator::new_prefixed, and that will fail with an Invalid state root error. (This worked previously without special-casing it like this because pairs just ignored the errors, and this new iterator doesn't do that anymore.)

H::default is [0; 32], that's an invalid trie root. Could have given the actual empty trie value but that's not convenient at all (makes the H depends on trie internals). In practice it 's just one step from saying any invalid root is an empty trie, but that would be confusing (like if you got a root and no content you cannot say if it is empty).
But yes that's rather awkward that some place allows H::default.

cheme · 2023-02-06T17:36:21Z

primitives/state-machine/src/ext.rs

+				});
+
+		if let Err(error) = result {
+			log::debug!(target: "trie", "Error while iterating the storage: {}", error);


Could be good to add a comment or insist on the fact that ignoring error is a host function related behavior. Don't really know how to word it.

cheme · 2023-02-06T17:43:32Z

client/rpc/src/state/state_full.rs

@@ -213,23 +213,29 @@ where
 			.map_err(client_err)
 	}

+	// TODO: This is horribly broken; either remove it, or make it streaming.


I think the paged variant is supposed to replace this rpc, so was planed to be removed.

Yep, I know. But technically we don't necessarily need to remove those non-paged variants. We could just fix them and make them stream the data. (It's out of scope of this PR though.)

cheme · 2023-02-06T17:44:36Z

client/db/src/record_stats_state.rs

+	type Backend = RecordStatsState<S, B>;
+	type Error = S::Error;
+
+	fn next_key(&mut self, backend: &Self::Backend) -> Option<Result<StorageKey, Self::Error>> {


Not for this PR, but could be good to record state (at least adding it looks straight forward).

primitives/state-machine/src/backend.rs

koute · 2023-02-14T14:38:39Z

@koute I just published trie-db v0.25.1 (small fix), is it possible to update your PR to use this version?

Sure thing.

cheme · 2023-02-14T14:45:12Z

Thanks a lot 🙏

cheme

I forgot to approve btw :) (IIRC only the question of he empty root was a bit bugging me but it is mainly me being finicky).

koute · 2023-02-14T14:46:58Z

@cheme Done.

Do you want to get this merged as soon as possible, or can it wait a little while?

As I've said, I wanted to run a full burn-in before merging this just to be on a safe side, but since running those takes time and is a pain I've planned to get my other storage iterator related changes finished too and run them in the same burn-in (even though I'm not going to push them to this PR) and only merge this once that passes. But if you want to get this merged sooner I can just start a burn-in only with these changes present.

cheme · 2023-02-14T14:48:13Z

@tomaka probably want it as soon as possible, but alternatively I can publish 0.24.1 with the fix.

koute · 2023-02-14T14:50:18Z

@tomaka probably want it as soon as possible, but alternatively I can publish 0.24.1 with the fix.

Okay; then let me start a burn in with only the changes from this PR then.

(Not sure if it matters but as a side effect this will also test your changes.)

…ge_iterators

koute · 2023-02-21T12:48:52Z

My burnin finished successfully, so this should be good to go I think.

paritytech-cicd-pr · 2023-02-21T12:58:22Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2423643

koute · 2023-02-22T07:49:16Z

bot merge

* Rework storage iterators * Make sure storage iteration is also accounted for when benchmarking * Use `trie-db` from crates.io * Appease clippy * Bump `trie-bench` to 0.35.0 * Fix tests' compilation * Update comment to clarify how `IterArgs::start_at` works * Add extra tests * Fix iterators on `Client` so that they behave as before * Add extra `unwrap`s in tests * More clippy fixes * Come on clippy, give me a break already * Rename `allow_missing` to `stop_on_incomplete_database` * Add `#[inline]` to `with_recorder_and_cache` * Use `with_recorder_and_cache` in `with_trie_db`; add doc comment * Simplify code: use `with_trie_db` in `next_storage_key_from_root` * Remove `expect`s in the benchmarking CLI * Add extra doc comments * Move `RawIter` before `TrieBackendEssence` (no code changes; just cut-paste) * Remove a TODO in tests * Update comment for `StorageIterator::was_complete` * Update `trie-db` to 0.25.1

koute added 2 commits January 31, 2023 19:58

Rework storage iterators

523ddf5

Make sure storage iteration is also accounted for when benchmarking

8497910

koute requested review from cheme, ggwpez and a team January 31, 2023 14:17

koute mentioned this pull request Jan 31, 2023

Companion for substrate#13284 paritytech/polkadot#6653

Merged

koute added 4 commits February 3, 2023 18:38

Use trie-db from crates.io

e6b35c9

Merge remote-tracking branch 'origin/master' into master_rework_stora…

6b640c8

…ge_iterators

Appease clippy

2556581

Bump trie-bench to 0.35.0

1f1ff7a

melekes reviewed Feb 3, 2023

View reviewed changes

client/db/src/bench.rs Show resolved Hide resolved

primitives/state-machine/src/trie_backend.rs Show resolved Hide resolved

primitives/state-machine/src/trie_backend.rs Show resolved Hide resolved

koute added 4 commits February 3, 2023 20:56

Fix tests' compilation

fa3edbd

Update comment to clarify how IterArgs::start_at works

4be0cad

Add extra tests

825ba75

Fix iterators on Client so that they behave as before

65d99ed

koute commented Feb 3, 2023

View reviewed changes

koute added 4 commits February 3, 2023 23:21

Add extra unwraps in tests

ab0fff6

More clippy fixes

2e41604

Come on clippy, give me a break already

b5ce5f5

Merge remote-tracking branch 'origin/master' into master_rework_stora…

104894b

…ge_iterators

melekes approved these changes Feb 6, 2023

View reviewed changes

client/api/src/backend.rs Show resolved Hide resolved

cheme reviewed Feb 6, 2023

View reviewed changes

Rename allow_missing to stop_on_incomplete_database

e049042

Update trie-db to 0.25.1

fdb6b63

cheme mentioned this pull request Feb 14, 2023

Merkle proof with unused entry #13357

Closed

2 tasks

cheme approved these changes Feb 14, 2023

View reviewed changes

Merge remote-tracking branch 'origin/master' into master_rework_stora…

fedc9f3

…ge_iterators

paritytech-processbot bot merged commit 86c6bb9 into paritytech:master Feb 22, 2023

This was referenced Feb 23, 2023

Further storage iterator refactoring #13445

Merged

Speed up storage iteration from within the runtime #13479

Merged

jacogr mentioned this pull request Mar 15, 2023

"Phantom" multisig approvals appearing on Accounts page polkadot-js/apps#9103

Open

jasl mentioned this pull request Mar 17, 2023

Upgrade to Polkadot v0.9.39 Phala-Network/phala-blockchain#1203

Merged

1xstj mentioned this pull request Mar 31, 2023

bump to polkadot-v0.9.39 tangle-network/tangle#156

Merged

5 tasks

kiltbot mentioned this pull request Apr 3, 2023

[AUTOMATIC] Update Polkadot dependencies from 0.9.38 to 0.9.39 KILTprotocol/kilt-node#495

Closed

librelois mentioned this pull request Apr 6, 2023

Update substrate/polkadot/cumulus from v0.9.38 to v0.9.40 moonbeam-foundation/moonbeam#2200

Closed

tomaka mentioned this pull request Apr 12, 2023

Parachain compatibility broken: UnusedProofEntry on token transfer smol-dot/smoldot#416

Closed

nbaztec mentioned this pull request Apr 25, 2023

Dependency (Substrate/Polkadot/Frontier/Cumulus/...) update to v0.9.40 moonbeam-foundation/moonbeam#2202

Merged

tomaka mentioned this pull request May 10, 2023

Error in call proof: UnusedProofEntry smol-dot/smoldot#567

Closed

kacperzuk-neti mentioned this pull request Jun 23, 2023

Polkadot v0.9.43 liberland/liberland_substrate#295

Merged

15 tasks

MOZGIII mentioned this pull request Nov 8, 2023

Tracking issue to bump substrate related deps to polkadot-v0.9.39 humanode-network/humanode#828

Closed

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework storage iterators #13284

Rework storage iterators #13284

koute commented Jan 31, 2023 •

edited

Loading

melekes left a comment

koute Feb 3, 2023

cheme Feb 3, 2023

koute Feb 3, 2023

melekes left a comment

cheme Feb 6, 2023

bkchr Feb 8, 2023

cheme Feb 6, 2023

koute Feb 13, 2023 •

edited

Loading

cheme Feb 13, 2023

cheme Feb 6, 2023

cheme Feb 6, 2023

koute Feb 7, 2023

cheme Feb 6, 2023

koute commented Feb 14, 2023

cheme commented Feb 14, 2023

cheme left a comment

koute commented Feb 14, 2023

cheme commented Feb 14, 2023

koute commented Feb 14, 2023

koute commented Feb 21, 2023

paritytech-cicd-pr commented Feb 21, 2023

koute commented Feb 22, 2023

Rework storage iterators #13284

Rework storage iterators #13284

Conversation

koute commented Jan 31, 2023 • edited Loading

Potential future work

melekes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

melekes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koute Feb 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koute commented Feb 14, 2023

cheme commented Feb 14, 2023

cheme left a comment

Choose a reason for hiding this comment

koute commented Feb 14, 2023

cheme commented Feb 14, 2023

koute commented Feb 14, 2023

koute commented Feb 21, 2023

paritytech-cicd-pr commented Feb 21, 2023

koute commented Feb 22, 2023

koute commented Jan 31, 2023 •

edited

Loading

koute Feb 13, 2023 •

edited

Loading