Faster sync #6563
Merged
Faster sync #6563
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
karlb
added a commit
to karlb/scenario-player
that referenced
this pull request
Oct 30, 2020
The selective event filtering introduced in raiden-network/raiden#6563 changed the API, which make this commit necessary.
fredo
approved these changes
Nov 4, 2020
raiden/blockchain/filters.py
Outdated
|
|
||
| def union(self, other: "RaidenContractFilter") -> "RaidenContractFilter": | ||
| """ Return a new RaidenContractFilter with all elements from both input filters """ | ||
| # We must not have to different non-None secret registries. Choose the non-None one. |
There was a problem hiding this comment.
Suggested change
| # We must not have to different non-None secret registries. Choose the non-None one. | |
| # We must not have two different non-None secret registries. Choose the non-None one. |
| @@ -318,21 +318,6 @@ def token_network_address(self) -> TokenNetworkAddress: | |||
| # FIXME: remove on next breaking release | |||
There was a problem hiding this comment.
Do we still need this comment?
There was a problem hiding this comment.
This (ContractReceiveRouteClosed) shouldn't be removed if the released is supposed to be backwards-compatible.
There was a problem hiding this comment.
right, but it is removed in this PR isn't it?
There was a problem hiding this comment.
That's my point. Depending on what the goals are, this should eventually stay.
Instead of requesting all events for the relevant contracts, we filter much more selectively by including the right topics.
Apparently, it is faster to include a topic, event when you want to filter for all events of a contract.
If we don't have a channel, we are not interested in any secrets registered online. Thus we can skip the slow event log queries until we open our first channel.
For each block range, we make multiple requests with different filters. Before this commit, only the last request duration has been used to adjust the batch size. But the point of the adjustment is to prevent any request from running into a timeout, so adjusting based on the maximal duration of all requests is a better way to achieve this.
After working with this a bit, the defaultdict seemed to make things less clear than a using a normal dict with `get` and `setdefault`.
karlb
added a commit
to karlb/scenario-player
that referenced
this pull request
Nov 5, 2020
The selective event filtering introduced in raiden-network/raiden#6563 changed the API, which make this commit necessary.
karlb
added a commit
to karlb/scenario-player
that referenced
this pull request
Nov 5, 2020
The selective event filtering introduced in raiden-network/raiden#6563 changed the API, which make this commit necessary.
palango
pushed a commit
to raiden-network/scenario-player
that referenced
this pull request
Nov 5, 2020
The selective event filtering introduced in raiden-network/raiden#6563 changed the API, which make this commit necessary.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Improve event syncing speed by filtering more selectively. This brings down the initial sync on mainnet for a fresh node to 30s (parity, infura) to 2mins (geth).
Before
Fetch all events for the relevant contracts and filter in the Raiden client
After
TokenNetworkCreatedevents from TN registryChannelOpenedevents that include the node as participantChannelClosed,ChannelSettled,ChannelNewDepositandChannelWithdraweventsSecretRevealedevents from the SecretRegistry once the node starts to participate in a channelCloses #6497
The SP integration test must fail until raiden-network/scenario-player#635 is merged, which has to be based on this PR. To break the cycle, I will force merge this when all tests but the SP integration are green.
Potential future optimizations if we need more speed: