feat(pruning): prune ChangeSets & History during pipeline #3728

joshieDo · 2023-07-11T19:06:23Z

Resolves #3433
Resolves #3434

Adds account and storage changeset pruning during execution stage (if set). If so, it will also prune during the AccountHistory and StorageHistory stages.
If pruning is enabled, hashing and merkle stages might have to be run from scratch on subsequent pipeline runs if there are not enough changesets. (more context)

Full pruning is not supported, and will throw error at reading the configuration. Minimum 64 block distance.

codecov · 2023-07-11T19:29:06Z

Codecov Report

Merging #3728 (c7d83ec) into main (1ac2f15) will increase coverage by 0.00%.
The diff coverage is 82.71%.

Files Changed	Coverage Δ
bin/reth/src/node/mod.rs	`11.89% <0.00%> (-0.10%)`	⬇️
bin/reth/src/stage/dump/merkle.rs	`0.00% <0.00%> (ø)`
bin/reth/src/stage/run.rs	`1.28% <0.00%> (-0.13%)`	⬇️
crates/primitives/src/lib.rs	`100.00% <ø> (ø)`
crates/prune/src/error.rs	`0.00% <ø> (ø)`
crates/stages/src/error.rs	`83.33% <ø> (ø)`
crates/primitives/src/prune/part.rs	`75.00% <33.33%> (-25.00%)`	⬇️
crates/stages/src/stages/hashing_account.rs	`97.87% <87.50%> (+0.08%)`	⬆️
crates/stages/src/stages/merkle.rs	`83.06% <87.87%> (+0.74%)`	⬆️
crates/stages/src/stages/hashing_storage.rs	`96.94% <90.47%> (+0.11%)`	⬆️
... and 7 more

... and 12 files with indirect coverage changes

Flag	Coverage Δ
integration-tests	`16.25% <0.00%> (-0.05%)`	⬇️
unit-tests	`64.22% <82.71%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
reth binary	`25.76% <0.00%> (-0.15%)`	⬇️
blockchain tree	`83.04% <ø> (ø)`
pipeline	`90.03% <95.67%> (+0.19%)`	⬆️
storage (db)	`74.30% <100.00%> (+0.02%)`	⬆️
trie	`94.70% <ø> (ø)`
txpool	`45.40% <ø> (-0.61%)`	⬇️
networking	`77.64% <ø> (-0.06%)`	⬇️
rpc	`58.52% <ø> (-0.02%)`	⬇️
consensus	`63.51% <ø> (ø)`
revm	`33.10% <ø> (ø)`
payload builder	`6.58% <ø> (ø)`
primitives	`88.05% <81.81%> (+0.11%)`	⬆️

joshieDo · 2023-07-17T10:55:19Z

since there was some overlap, should #3733 be pushed first

shekhirin

LGTM overall, nits. Appreciate the tests!

crates/primitives/src/prune/target.rs

crates/stages/src/stages/index_account_history.rs

crates/stages/src/stages/mod.rs

crates/stages/src/stages/index_account_history.rs

…sets pruned

joshieDo · 2023-07-28T10:42:01Z

With this PR, lower end machines might not be able to catch up if their pruning is very aggressive. This happens because the pipeline re-run that happens after the first sync will trigger the hashing and merkle stages to be run from scratch since there might not be enough changesets. And if it cannot finish before 64 blocks have been merged, this behaviour will repeat itself.

Follow-up PR will handle that by temporarily not pruning so many changesets, so we can leverage the incremental branches of these stages. But it requires a few too many changes for this PR

mattsse

nit re duplicate condition,

otherwise lgtm

crates/primitives/src/prune/part.rs

mattsse · 2023-07-28T10:56:54Z

crates/primitives/src/prune/part.rs


 /// Part of the data that can be pruned.
 #[main_codec]
-#[derive(Debug, Clone, Copy, Eq, PartialEq, Ord, PartialOrd)]
+#[derive(Debug, Display, Clone, Copy, Eq, PartialEq, Ord, PartialOrd)]
 pub enum PrunePart {


@shekhirin what do you think about renaming this to PruneStep?

Personally, "Step" sounds like a consecutive action like "Stage" in the context of pipeline, while in pruning you can enable different parts and it doesn't really matter in which order and composition they are executed. But I do agree that "Part" is weird too.

maybe PruneComponent?

mattsse · 2023-07-28T10:59:35Z

crates/stages/src/stages/hashing_account.rs

+        if to_block - from_block > self.clean_threshold || from_block == 1 || !has_enough_changesets
+        {


we have the same check twice,

is it worth it to move this to a separate function and properly document it?

joshieDo added 4 commits July 11, 2023 18:23

prune changesets when writing to db from post state

90ae3c3

prune history on their stages

e980b15

add deserializer helper to acc/storage history pruning

1899c85

add deserialize_opt_prune_mode_unsupported_full test

2f05db3

joshieDo changed the title ~~pruning: prune ChangeSets & History during pipeline~~ feat(pruning): prune ChangeSets & History during pipeline Jul 11, 2023

joshieDo added C-enhancement New feature or request A-db Related to the database labels Jul 11, 2023

joshieDo added 5 commits July 15, 2023 14:20

Merge remote-tracking branch 'origin/main' into pruning/hist

880d2f6

add changeset checks to execution prune test

5466e9b

add history stages to the prune test

1ba88ba

move test_prune

b4b013e

rename to gen_prune_methods

87f475e

joshieDo marked this pull request as ready for review July 16, 2023 16:47

joshieDo requested review from rakita, onbjerg, rkrasiuk, shekhirin and gakonst as code owners July 16, 2023 16:47

joshieDo marked this pull request as draft July 25, 2023 18:21

joshieDo added 7 commits July 25, 2023 18:51

Merge remote-tracking branch 'origin/main' into pruning/hist

5b41c0d

dedup test struct

bf93292

fix prune_target_block when PruneMode::Before

f0cf8f6

prune_target_block_ only returns None on invalid configuration

2539015

history prune parts require 64 min blocks

e30447d

fix up test_prune

1e96152

add an explanation to test_prune

792ef89

joshieDo marked this pull request as ready for review July 25, 2023 22:36

remove dup test

d451ddf

joshieDo added 3 commits July 26, 2023 11:43

prune_target_block_* returns err if badly configured

c89a8da

revert serde_helper change

c1fcc86

add missing doc

e380fff

shekhirin approved these changes Jul 26, 2023

View reviewed changes

joshieDo added 3 commits July 26, 2023 17:35

rename prune_targets to prune_modes

2e707df

add explanation on test_prune

b4ff2d1

check if target has been reached after prune check on history stages

28d6f67

joshieDo added this pull request to the merge queue Jul 26, 2023

joshieDo removed this pull request from the merge queue due to a manual request Jul 26, 2023

joshieDo added 5 commits July 27, 2023 18:09

run from scratch hashing and merkle stages if there's required change…

a483c8b

…sets pruned

remove dbg line

d46a5bc

fix stage test runners

6a33867

Merge remote-tracking branch 'origin/main' into pruning/hist

84b0f27

fix testrunner

0e88fc9

mattsse approved these changes Jul 28, 2023

View reviewed changes

Merge remote-tracking branch 'origin/main' into pruning/hist

c7d83ec

joshieDo added this pull request to the merge queue Jul 31, 2023

Merged via the queue into main with commit 134fe81 Jul 31, 2023

joshieDo deleted the pruning/hist branch July 31, 2023 14:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(pruning): prune ChangeSets & History during pipeline #3728

feat(pruning): prune ChangeSets & History during pipeline #3728

joshieDo commented Jul 11, 2023 •

edited

Loading

codecov bot commented Jul 11, 2023 •

edited

Loading

joshieDo commented Jul 17, 2023

shekhirin left a comment •

edited

Loading

joshieDo commented Jul 28, 2023 •

edited

Loading

mattsse left a comment

mattsse Jul 28, 2023

shekhirin Jul 28, 2023

shekhirin Jul 28, 2023

mattsse Jul 28, 2023

		if to_block - from_block > self.clean_threshold \|\| from_block == 1 \|\| !has_enough_changesets
		{

feat(pruning): prune ChangeSets & History during pipeline #3728

feat(pruning): prune ChangeSets & History during pipeline #3728

Conversation

joshieDo commented Jul 11, 2023 • edited Loading

codecov bot commented Jul 11, 2023 • edited Loading

Codecov Report

joshieDo commented Jul 17, 2023

shekhirin left a comment • edited Loading

Choose a reason for hiding this comment

joshieDo commented Jul 28, 2023 • edited Loading

mattsse left a comment

Choose a reason for hiding this comment

mattsse Jul 28, 2023

Choose a reason for hiding this comment

shekhirin Jul 28, 2023

Choose a reason for hiding this comment

shekhirin Jul 28, 2023

Choose a reason for hiding this comment

mattsse Jul 28, 2023

Choose a reason for hiding this comment

joshieDo commented Jul 11, 2023 •

edited

Loading

codecov bot commented Jul 11, 2023 •

edited

Loading

shekhirin left a comment •

edited

Loading

joshieDo commented Jul 28, 2023 •

edited

Loading