Lock the state transition during conversion #384
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The problem still seems to occur, which would point to the state not being properly saved at the right time instead of a pure conversion issue - or something that is not covered by the lock. Attaching relevant logs here for future reference. |
|
The new update seems to be performing much better, and in fact is so far on par with the results before the rebase. Over a quarter of the transition happened so far. |
* add conversion test to workflow * mandatory -f switch fix in rm * forgot & at the end of the geth command * remove incorrect kill * add debug traces * add an overlay stride * fix typo
|
it achieved a shadowfork, it's definitely good to go. |
gballet
added a commit
that referenced
this pull request
Feb 22, 2024
* heavy handed approach: lock the state transition during conversion * also lock transition state loading/unloading * reduce logs verbosity * add conversion test to workflow (#386) * add conversion test to workflow * mandatory -f switch fix in rm * forgot & at the end of the geth command * remove incorrect kill * add debug traces * add an overlay stride * fix typo
gballet
added a commit
that referenced
this pull request
Feb 22, 2024
…tion states (#375) * persist conversion state to db * fix: don't recreate LRU when writing state * opt: only write state to db if not already present in LRU * fix: rlp can't encode TransitionState * fix: use gob because binary.Write does not support structs 🤦♂️ * fix: nil pointer panic * add logs to debug shadowfork * no such thing as not enough traces * ditto * fix stupid bug * add a comment for readability * add more traces * Lock the state transition during conversion (#384) * heavy handed approach: lock the state transition during conversion * also lock transition state loading/unloading * reduce logs verbosity * add conversion test to workflow (#386) * add conversion test to workflow * mandatory -f switch fix in rm * forgot & at the end of the geth command * remove incorrect kill * add debug traces * add an overlay stride * fix typo * Apply suggestions from code review
gballet
added a commit
that referenced
this pull request
Feb 28, 2024
…tion states (#375) * persist conversion state to db * fix: don't recreate LRU when writing state * opt: only write state to db if not already present in LRU * fix: rlp can't encode TransitionState * fix: use gob because binary.Write does not support structs 🤦♂️ * fix: nil pointer panic * add logs to debug shadowfork * no such thing as not enough traces * ditto * fix stupid bug * add a comment for readability * add more traces * Lock the state transition during conversion (#384) * heavy handed approach: lock the state transition during conversion * also lock transition state loading/unloading * reduce logs verbosity * add conversion test to workflow (#386) * add conversion test to workflow * mandatory -f switch fix in rm * forgot & at the end of the geth command * remove incorrect kill * add debug traces * add an overlay stride * fix typo * Apply suggestions from code review
gballet
added a commit
that referenced
this pull request
Mar 26, 2024
…tion states (#375) * persist conversion state to db * fix: don't recreate LRU when writing state * opt: only write state to db if not already present in LRU * fix: rlp can't encode TransitionState * fix: use gob because binary.Write does not support structs 🤦♂️ * fix: nil pointer panic * add logs to debug shadowfork * no such thing as not enough traces * ditto * fix stupid bug * add a comment for readability * add more traces * Lock the state transition during conversion (#384) * heavy handed approach: lock the state transition during conversion * also lock transition state loading/unloading * reduce logs verbosity * add conversion test to workflow (#386) * add conversion test to workflow * mandatory -f switch fix in rm * forgot & at the end of the geth command * remove incorrect kill * add debug traces * add an overlay stride * fix typo * Apply suggestions from code review
gballet
added a commit
that referenced
this pull request
Apr 15, 2024
…tion states (#375) * persist conversion state to db * fix: don't recreate LRU when writing state * opt: only write state to db if not already present in LRU * fix: rlp can't encode TransitionState * fix: use gob because binary.Write does not support structs 🤦♂️ * fix: nil pointer panic * add logs to debug shadowfork * no such thing as not enough traces * ditto * fix stupid bug * add a comment for readability * add more traces * Lock the state transition during conversion (#384) * heavy handed approach: lock the state transition during conversion * also lock transition state loading/unloading * reduce logs verbosity * add conversion test to workflow (#386) * add conversion test to workflow * mandatory -f switch fix in rm * forgot & at the end of the geth command * remove incorrect kill * add debug traces * add an overlay stride * fix typo * Apply suggestions from code review
gballet
added a commit
that referenced
this pull request
Apr 29, 2024
quell linter issue support Transition post tree in conversion Refactor transition post genesis (#311) * rewrite per-block conversion pointer management * remove unused method * fix: a branch that can verge at genesis or post genesis (#314) * fix: import cycle in conversion refactor (#315) * fix shadowfork panic in OpenStorageTrie fix OpenStorageTrie: return an error instead of panicking if the account tree not a verkle tree (#321) add a switch to force proof in blocks (#322) * add a switch to force proof in blocks * activate switch * fix switch type add switch to override the stride of the overlay conversion (#323) * add switch to override the stride of the overlay conversion * set a default stride of 10k add a few traces for relaunch add missing step in the chain of setting the override for overlay stride fix: save and load transition state for block processing (#324) * fix: save and load transition state for block processing * log whether the tree is verkle in LoadTransitionState * fix: ensure the transition is marked as started in insertChain * dump saved address * fix nil pointer panic * remove stacktrace that is no longer useful * fix a panic * fix build * check: copy current account address BEFORE it's saved * mandatory panic fix * Remove debug fmt.Println * more cleanup + comments fix: ensure StorageProcessed is properly recovered add traces for gas pool overflow cmd/{geth, utils}: add a command to clear verkle costs (#326) * cmd/{geth, utils}: add a command to clear verkle costs fix: boolean issue fix: load finalization state in FinalizeAndAssemble (#340) * Conversion and TransitionTrie fixes (#346) * fixes Signed-off-by: Ignacio Hagopian <[email protected]> * remove old comment Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * trace cleanup --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix: check for nil override in NewBlockChain (#357) fix rebase issue fix a few rebase issues add debug traces fix: assume that having to create a new transaction state mean it hasn't happened. This is a workaround, because it will always be false when restarting a node that has started the transition but not necessarily completed it. add logs to debug shadowfork persist conversion state to db and use an LRU cache for active transition states (#375) * persist conversion state to db * fix: don't recreate LRU when writing state * opt: only write state to db if not already present in LRU * fix: rlp can't encode TransitionState * fix: use gob because binary.Write does not support structs 🤦♂️ * fix: nil pointer panic * add logs to debug shadowfork * no such thing as not enough traces * ditto * fix stupid bug * add a comment for readability * add more traces * Lock the state transition during conversion (#384) * heavy handed approach: lock the state transition during conversion * also lock transition state loading/unloading * reduce logs verbosity * add conversion test to workflow (#386) * add conversion test to workflow * mandatory -f switch fix in rm * forgot & at the end of the geth command * remove incorrect kill * add debug traces * add an overlay stride * fix typo * Apply suggestions from code review eth: add debug_conversionStatus RPC call (#392) * eth: add debug_conversionStatus RPC call * add debug trace4s * Apply suggestions from code review * export started/ended fields fix post-verge sync (#404) * fix post-verge sync * review: fix truncated comment
gballet
added a commit
that referenced
this pull request
Apr 29, 2024
quell linter issue support Transition post tree in conversion Refactor transition post genesis (#311) * rewrite per-block conversion pointer management * remove unused method * fix: a branch that can verge at genesis or post genesis (#314) * fix: import cycle in conversion refactor (#315) * fix shadowfork panic in OpenStorageTrie fix OpenStorageTrie: return an error instead of panicking if the account tree not a verkle tree (#321) add a switch to force proof in blocks (#322) * add a switch to force proof in blocks * activate switch * fix switch type add switch to override the stride of the overlay conversion (#323) * add switch to override the stride of the overlay conversion * set a default stride of 10k add a few traces for relaunch add missing step in the chain of setting the override for overlay stride fix: save and load transition state for block processing (#324) * fix: save and load transition state for block processing * log whether the tree is verkle in LoadTransitionState * fix: ensure the transition is marked as started in insertChain * dump saved address * fix nil pointer panic * remove stacktrace that is no longer useful * fix a panic * fix build * check: copy current account address BEFORE it's saved * mandatory panic fix * Remove debug fmt.Println * more cleanup + comments fix: ensure StorageProcessed is properly recovered add traces for gas pool overflow cmd/{geth, utils}: add a command to clear verkle costs (#326) * cmd/{geth, utils}: add a command to clear verkle costs fix: boolean issue fix: load finalization state in FinalizeAndAssemble (#340) * Conversion and TransitionTrie fixes (#346) * fixes Signed-off-by: Ignacio Hagopian <[email protected]> * remove old comment Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * trace cleanup --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix: check for nil override in NewBlockChain (#357) fix rebase issue fix a few rebase issues add debug traces fix: assume that having to create a new transaction state mean it hasn't happened. This is a workaround, because it will always be false when restarting a node that has started the transition but not necessarily completed it. add logs to debug shadowfork persist conversion state to db and use an LRU cache for active transition states (#375) * persist conversion state to db * fix: don't recreate LRU when writing state * opt: only write state to db if not already present in LRU * fix: rlp can't encode TransitionState * fix: use gob because binary.Write does not support structs 🤦♂️ * fix: nil pointer panic * add logs to debug shadowfork * no such thing as not enough traces * ditto * fix stupid bug * add a comment for readability * add more traces * Lock the state transition during conversion (#384) * heavy handed approach: lock the state transition during conversion * also lock transition state loading/unloading * reduce logs verbosity * add conversion test to workflow (#386) * add conversion test to workflow * mandatory -f switch fix in rm * forgot & at the end of the geth command * remove incorrect kill * add debug traces * add an overlay stride * fix typo * Apply suggestions from code review eth: add debug_conversionStatus RPC call (#392) * eth: add debug_conversionStatus RPC call * add debug trace4s * Apply suggestions from code review * export started/ended fields fix post-verge sync (#404) * fix post-verge sync * review: fix truncated comment
Closed
gballet
added a commit
that referenced
this pull request
May 2, 2024
quell linter issue support Transition post tree in conversion Refactor transition post genesis (#311) * rewrite per-block conversion pointer management * remove unused method * fix: a branch that can verge at genesis or post genesis (#314) * fix: import cycle in conversion refactor (#315) * fix shadowfork panic in OpenStorageTrie fix OpenStorageTrie: return an error instead of panicking if the account tree not a verkle tree (#321) add a switch to force proof in blocks (#322) * add a switch to force proof in blocks * activate switch * fix switch type add switch to override the stride of the overlay conversion (#323) * add switch to override the stride of the overlay conversion * set a default stride of 10k add a few traces for relaunch add missing step in the chain of setting the override for overlay stride fix: save and load transition state for block processing (#324) * fix: save and load transition state for block processing * log whether the tree is verkle in LoadTransitionState * fix: ensure the transition is marked as started in insertChain * dump saved address * fix nil pointer panic * remove stacktrace that is no longer useful * fix a panic * fix build * check: copy current account address BEFORE it's saved * mandatory panic fix * Remove debug fmt.Println * more cleanup + comments fix: ensure StorageProcessed is properly recovered add traces for gas pool overflow cmd/{geth, utils}: add a command to clear verkle costs (#326) * cmd/{geth, utils}: add a command to clear verkle costs fix: boolean issue fix: load finalization state in FinalizeAndAssemble (#340) * Conversion and TransitionTrie fixes (#346) * fixes Signed-off-by: Ignacio Hagopian <[email protected]> * remove old comment Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * trace cleanup --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix: check for nil override in NewBlockChain (#357) fix rebase issue fix a few rebase issues add debug traces fix: assume that having to create a new transaction state mean it hasn't happened. This is a workaround, because it will always be false when restarting a node that has started the transition but not necessarily completed it. add logs to debug shadowfork persist conversion state to db and use an LRU cache for active transition states (#375) * persist conversion state to db * fix: don't recreate LRU when writing state * opt: only write state to db if not already present in LRU * fix: rlp can't encode TransitionState * fix: use gob because binary.Write does not support structs 🤦♂️ * fix: nil pointer panic * add logs to debug shadowfork * no such thing as not enough traces * ditto * fix stupid bug * add a comment for readability * add more traces * Lock the state transition during conversion (#384) * heavy handed approach: lock the state transition during conversion * also lock transition state loading/unloading * reduce logs verbosity * add conversion test to workflow (#386) * add conversion test to workflow * mandatory -f switch fix in rm * forgot & at the end of the geth command * remove incorrect kill * add debug traces * add an overlay stride * fix typo * Apply suggestions from code review eth: add debug_conversionStatus RPC call (#392) * eth: add debug_conversionStatus RPC call * add debug trace4s * Apply suggestions from code review * export started/ended fields fix post-verge sync (#404) * fix post-verge sync * review: fix truncated comment
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a heavy-handed approach compared to #382, but I hope that it would cause less problems. If this is confirmed to work, it will then make sense to try a more fine-grained approach.