Support MMR Pruning #9700

AurevoirXavier · 2021-09-06T00:24:31Z

Add pruning process to the MMR pallet.

But I have encountered a problem with the verify_leaf part.
should_verify_on_the_next_block_since_there_is_no_pruning_yet this failed.
We can't get the root at some moment (except the latest one) if we perform pruning on the Nodes.

@tomusdrw Any idea on that?

Closes paritytech/grandpa-bridge-gadget#64

hackfisher · 2021-09-06T07:48:30Z

But I have encountered a problem with the verify_leaf part.
should_verify_on_the_next_block_since_there_is_no_pruning_yet this failed.
We can't get the root at some moment (except the latest one) if we perform pruning on the Nodes.

After support pruning, ModuleMmr<mmr::storage::RuntimeStorage, T, I> can only be used to verify proof generated agains latest root, proof generated agains previous roots(and its leaves) will require previous roots(or leaves) for verification.

In this test, root in runtime storage has already been changed, and leaves for generating that root have already been pruned after new_block();.

OffchainStorage might be need to be used for generating and verifying proof.

tomusdrw

Cool stuff, thanks!

I think the idea of removing nodes on every insertion is sensible - it's going to give us much more predictable on_initialize weight compared to defered pruning.

We have to be careful though, to asses what the actual weight is going to be. The computation time is now (slowly) growing depending on the MMR size (more/less peaks), so the formula and benchmarking needs to be updated accordingly.

tomusdrw · 2021-09-07T10:32:43Z

frame/merkle-mountain-range/src/mmr/storage.rs

+		let elems = elems
+			.into_iter()
+			.enumerate()
+			.map(|(i, elem)| (size + i as NodeIndex, elem))
+			.collect::<Vec<_>>();
+		let leaves = elems.iter().fold(leaves, |acc, (pos, elem)| {
 			// Indexing API is used to store the full leaf content.
-			let key = Pallet::<T, I>::offchain_key(size);
-			elem.using_encoded(|elem| sp_io::offchain_index::set(&key, elem));
-			size += 1;
+			elem.using_encoded(|elem| {
+				offchain_index::set(&Pallet::<T, I>::offchain_key(*pos), elem)
+			});

 			if let Node::Data(..) = elem {
-				leaves += 1;
+				acc + 1
+			} else {
+				acc
 			}
-		}
+		});


These changes seem unnecessary and imho increase both code complexity and the number of iterations + extra allocation.

I don't like mutable variable. IMO this simplify the code.
But yes, a little bit more allocation here. Could you accept this?

frame/merkle-mountain-range/src/mmr/storage.rs

AurevoirXavier · 2021-09-10T08:31:33Z

Cool stuff, thanks!

I think the idea of removing nodes on every insertion is sensible - it's going to give us much more predictable on_initialize weight compared to defered pruning.

We have to be careful though, to asses what the actual weight is going to be. The computation time is now (slowly) growing depending on the MMR size (more/less peaks), so the formula and benchmarking needs to be updated accordingly.

How do we benchmark this?

frame/merkle-mountain-range/src/mmr/storage.rs

AurevoirXavier · 2021-09-13T13:46:04Z

@tomusdrw Please review.

AurevoirXavier · 2021-09-23T06:44:58Z

@tomusdrw We really need this feature! 🙏🏻

tomusdrw · 2021-09-27T13:33:48Z

@AurevoirXavier apologies for the delay, reviewing this PR is high on my list, hoping to get this done this week.

AurevoirXavier · 2021-10-25T05:59:39Z

@AurevoirXavier apologies for the delay, reviewing this PR is high on my list, hoping to get this done this week.

@tomusdrw Any progress?

AurevoirXavier · 2021-11-16T13:36:45Z

@adoerr Could this be merged?

adoerr · 2021-11-16T13:38:24Z

@tomusdrw is giving it another look right now

tomusdrw · 2021-11-17T23:02:54Z

@tomusdrw is giving it another look right now

Proposed my changes here: darwinia-network#123
Main ares of focus:

Distinction between NodeIndex and LeafIndex types to avoid confusion
Code reduction, removing extra loops and allocations

Simplification of insertion & pruning code

tomusdrw · 2021-11-18T15:42:01Z

@AurevoirXavier could you run cargo fmt to make the CI happy?

frame/merkle-mountain-range/primitives/src/lib.rs

tomusdrw · 2021-11-23T13:03:36Z

bot merge

paritytech-processbot · 2021-11-23T13:03:45Z

Waiting for commit status.

AurevoirXavier · 2021-11-23T14:21:12Z

Just a note here, the weights were not getting updated.

@tomusdrw

Could you mention me while updating the weights?
I'm very interested in how to bench this.

* Use `0.3.2` * Replace `u64` with `NodeIndex` * Fix Typo * Add Pruning Logic * Fix Some Tests * Remove Comment * Log Only Under STD * Return while No Element to Append * Optimize Pruning Algorithm * Update Doc * Update Doc * Zero Copy Algorithm * Import Missing Type * Fix Merge Mistake * Import Missing Item * Make `verify` Off-Chain * `cargo fmt` * Avoid using NodeIndex in incorrect places. * Simplify pruning. * Format Co-authored-by: Tomasz Drwięga <[email protected]>

Xavier Lau added 5 commits September 6, 2021 06:30

Use 0.3.2

d99bb52

Replace u64 with NodeIndex

5cb4a96

Fix Typo

f84ccda

Add Pruning Logic

fbc7804

Fix Some Tests

3c748e7

AurevoirXavier mentioned this pull request Sep 6, 2021

Parity MMR darwinia-network/darwinia-common#743

Closed

bkchr requested a review from tomusdrw September 6, 2021 07:26

bkchr added A0-please_review Pull request needs code review. B7-runtimenoteworthy labels Sep 6, 2021

bkchr added the C1-low PR touches the given topic and has a low impact on builders. label Sep 6, 2021

tomusdrw reviewed Sep 7, 2021

View reviewed changes

Xavier Lau added 2 commits September 9, 2021 11:29

Remove Comment

687eab7

Log Only Under STD

1cbe99a

AurevoirXavier force-pushed the xavier-mmr-pruning branch from 47c5cbd to 1cbe99a Compare September 9, 2021 03:39

Xavier Lau added 4 commits September 9, 2021 13:34

Return while No Element to Append

6b71e32

Optimize Pruning Algorithm

15b5a3f

Update Doc

32207f1

Update Doc

7ccee87

AurevoirXavier commented Sep 11, 2021

View reviewed changes

frame/merkle-mountain-range/src/mmr/storage.rs Outdated Show resolved Hide resolved

Zero Copy Algorithm

7028381

AurevoirXavier mentioned this pull request Sep 13, 2021

Refactor and migrate darwinia mmr pallet to use substrate mmr pallet darwinia-network/darwinia-common#623

Closed

Merge from Master

95849c3

adoerr self-requested a review November 11, 2021 06:51

Xavier Lau added 2 commits November 16, 2021 13:29

Make verify Off-Chain

58fb0a2

cargo fmt

998cc36

AurevoirXavier requested a review from andresilva as a code owner November 16, 2021 06:35

Avoid using NodeIndex in incorrect places.

6ea225c

Simplify pruning.

7b580bb

Merge pull request #123 from tomusdrw/td-mmr-pruning

a6903e1

Simplification of insertion & pruning code

tomusdrw approved these changes Nov 18, 2021

View reviewed changes

adoerr self-requested a review November 18, 2021 16:06

adoerr approved these changes Nov 18, 2021

View reviewed changes

frame/merkle-mountain-range/primitives/src/lib.rs Show resolved Hide resolved

Format

2b7f205

tomusdrw added the D2-notlive 💤 PR contains changes in a runtime directory that is not deployed to a chain that requires an audit. label Nov 23, 2021

paritytech-processbot bot merged commit 7a531b0 into paritytech:master Nov 23, 2021

tomusdrw mentioned this pull request Nov 26, 2021

Update on_initialize weight calculations paritytech/grandpa-bridge-gadget#351

Closed

github-actions bot mentioned this pull request Dec 2, 2021

Update substrate/polkadot/cumulus from v0.9.12 to v0.9.13 moonbeam-foundation/moonbeam#1050

Closed

ghzlatarev mentioned this pull request Dec 6, 2021

[Manta] Integrate v0.9.13 upstream changes Manta-Network/Manta#294

Merged

23 tasks

AurevoirXavier deleted the xavier-mmr-pruning branch March 2, 2022 06:42

AurevoirXavier mentioned this pull request Sep 28, 2022

Member request polkadot-fellows/seeding#12

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support MMR Pruning #9700

Support MMR Pruning #9700

AurevoirXavier commented Sep 6, 2021 •

edited by tomusdrw

Loading

hackfisher commented Sep 6, 2021 •

edited

Loading

tomusdrw left a comment

tomusdrw Sep 7, 2021

AurevoirXavier Sep 9, 2021 •

edited

Loading

AurevoirXavier commented Sep 10, 2021

AurevoirXavier commented Sep 13, 2021

AurevoirXavier commented Sep 23, 2021

tomusdrw commented Sep 27, 2021

AurevoirXavier commented Oct 25, 2021 •

edited

Loading

AurevoirXavier commented Nov 16, 2021

adoerr commented Nov 16, 2021

tomusdrw commented Nov 17, 2021

tomusdrw commented Nov 18, 2021

tomusdrw commented Nov 23, 2021

paritytech-processbot bot commented Nov 23, 2021

AurevoirXavier commented Nov 23, 2021 •

edited

Loading

Support MMR Pruning #9700

Support MMR Pruning #9700

Conversation

AurevoirXavier commented Sep 6, 2021 • edited by tomusdrw Loading

hackfisher commented Sep 6, 2021 • edited Loading

tomusdrw left a comment

Choose a reason for hiding this comment

tomusdrw Sep 7, 2021

Choose a reason for hiding this comment

AurevoirXavier Sep 9, 2021 • edited Loading

Choose a reason for hiding this comment

AurevoirXavier commented Sep 10, 2021

AurevoirXavier commented Sep 13, 2021

AurevoirXavier commented Sep 23, 2021

tomusdrw commented Sep 27, 2021

AurevoirXavier commented Oct 25, 2021 • edited Loading

AurevoirXavier commented Nov 16, 2021

adoerr commented Nov 16, 2021

tomusdrw commented Nov 17, 2021

tomusdrw commented Nov 18, 2021

tomusdrw commented Nov 23, 2021

paritytech-processbot bot commented Nov 23, 2021

AurevoirXavier commented Nov 23, 2021 • edited Loading

AurevoirXavier commented Sep 6, 2021 •

edited by tomusdrw

Loading

hackfisher commented Sep 6, 2021 •

edited

Loading

AurevoirXavier Sep 9, 2021 •

edited

Loading

AurevoirXavier commented Oct 25, 2021 •

edited

Loading

AurevoirXavier commented Nov 23, 2021 •

edited

Loading