-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve speed of nonconsensus data removal #2717
Conversation
Problem: removal of nonconsensus data is too inefficient and as a result blocks are imported too slow. Solution: reformulation of deletion logic for better performance
Pull Request Test Coverage Report for Build 16282840-fb5b-4d95-9aba-e9af732f0d66
💛 - Coveralls |
{_, result} = | ||
acquire_query = | ||
from( | ||
block in where_invalid_neighbour(changes_list), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PR's description says
This PR changes the logic to remove only data of blocks that lose consensus because they are being overridden (as opposed to also for blocks that are invalid **neighbors**)
it looks like it also removes data from invalid neighbours because they are selected with where_invalid_neighbour(changes_list)
I see optimisation in using block_hash
'es for selecting transactions instread of using block_number
's because we have DB index for block_hash
in the transactions
table
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ayrat555 yes the invalid neighbors blocks are still removed, however if you look at each remove_nonconsensus_xxx
step you'll find that now they take transaction hashes from the result of derive_transaction_forks
, in other terms all the hashes of the transactions that have been forked.
What I figured was in fact that we do not fork transactions for invalid neighbors because those will be handled after anyway (neighbor loses consensus > get refetched > new block is inserted and old transactions get forked) and we can do the same for the rest of the nonconsensus data.
Motivation
The recently introduction of non-consensus block's data removal is not efficient enough to keep the pace with block creation.
Changelog
Enhancements
This PR changes the logic to remove only data of blocks that lose consensus because they are being overridden (as opposed to also for blocks that are invalid neighbors) and changes the queries to perform.
Checklist for your PR
CHANGELOG.md
with this PRmaster
in theVersion
column.