[Processor] Use multiple threads to speed up FA processor. #460
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rtso
reviewed
Aug 16, 2024
Comment on lines
+763
to
+768
| for (faa, fab, acs, cfab, fam) in data { | ||
| fungible_asset_activities.extend(faa); | ||
| fungible_asset_balances.extend(fab); | ||
| all_coin_supply.extend(acs); | ||
| current_fungible_asset_balances.extend(cfab); | ||
| fungible_asset_metadata.extend(fam); |
There was a problem hiding this comment.
I don't want to change the insert_to_db interface for now, today we want to give it a larger batch to get better performance.
rtso
approved these changes
Aug 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Most part of this PR is just from auto-formatter.
This PR is trying to use rayon to process transactions in multiple threads. I moved all the hashmaps inside the loop (so assumption here is everything txn can be processed independently), and produces per txn output. The output is merged together at the very end.
The performance improvement depends on the machine spec. On my dev machine processing p2p transfer data is 10x+ faster.