Execute all address' transactions page queries in parallel #2636
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pasqu4le
force-pushed
the
pp-parallel-address-transactions
branch
2 times, most recently
from
7fd51b7
to
813857a
Compare
Pull Request Test Coverage Report for Build 0952427c-0c7b-4e2f-bbd6-deecf1c87858
💛 - Coveralls |
pasqu4le
force-pushed
the
pp-parallel-address-transactions
branch
from
813857a
to
dbf760c
Compare
pasqu4le
force-pushed
the
pp-parallel-address-transactions
branch
from
f2c0385
to
0aa3eda
Compare
pasqu4le
added
ready for review
ayrat555
reviewed
Aug 27, 2019
|
|
||
| defp wait_for_address_transactions(tasks) do | ||
| tasks | ||
| |> Task.yield_many(:timer.seconds(20)) |
There was a problem hiding this comment.
are you sure they will be finished in 20 seconds? did you test on slow pages?
There was a problem hiding this comment.
I am not, I think we should try on staging to be sure.
I did test on a couple of addresses with many transactions only.
But more then that, if we make it bigger we'd hit the page timeout anyway, right?
ayrat555
approved these changes
Aug 29, 2019
vbaranov
approved these changes
Sep 3, 2019
There was a problem hiding this comment.
I applied this patch to stg. And It definitely loads transactions much faster for address page even when address has ~500 000 txs. For instance, for this address page ./address/0x5e032243d507c743b061ef021e2ec7fcc6d3ab89/transactions the first page with the list of transactions is loaded in several seconds whereas on prod instance it failed with timeout error
ayrat555
added a commit
that referenced
this pull request
Sep 3, 2019
In this PR (#2636) we started fetching transactions and rewards in parallel using elixir tasks. This PR uses similar approach to fetch validation and transaction counters on address page.
DanielGolds
pushed a commit
to responsible-gold/blockscout
that referenced
this pull request
Sep 25, 2019
* fix env var description * checkout package-lock.json from origin/master * Re-added internationalization, re-checked all changes on all devices * Re-added internationalization, re-checked all changes on all devices, resolved merge conflict * Add scheme in api urls * Move api's url to the end of example * Use pipeline * Added upstreamrepo for local repo,fetched updates, re-added internalization * Added Xerom to list of Additional Chains using BlockScout. * Added PR * Fixed incorrect numerical order in Chores * Fetched last updates from forked repo * Changelog.md Clean up * README.md Clean up * Update README.md * Fixed tooltips issue, header should work fine, fixed buttons colors on Contract Address Details page * Changes staging * Changed text color on top miner address tooltip * Set not full path to api on ajax queries * Tried to resolve gettext issue again * Removed unnecessary lines. * Delete _erc721_token_image_container.scss * Update _dashboard-banner.scss * Update overview.html.eex * Update _details.html.eex * Update index.html.eex * Gettext issue should be fixed * Fixed buttons color at smart contract section * Update README.md LUKSO L14 testnet in the list of hosted chains * fix market_cap calculation * fix tests * add fallback * check if blockscout comment is already added * update gettext * add default case * add CHANGELOG entry * fix gettext * fix tests * fix remaining tests * Revert some unintentional changes * Additional cleanup * gettext fix * Cleanup _dashboard-banner.scss * Add CHANGELOG entry * Remove unnecessary whitespaces * set correct last calue for coin balances chart data Sometimes the last value for coin history chart data is incorrect because when selecting records they are grouped by day and max value is selected in that day. So if there are a couple of coin balance changes only max value is shown. This PR always shows the latest value from the database by block number. * add CHANGELOG entry * fix test * fix tests * fetch coin gecko id based on coin symbol Coin Gecko provides API with all listed coins. We can use this API to find coin id instead of setting it by hand. * add CHANGELOG entry * remove config entry * find index of the comment instead of item * fetch address counters async In this PR (blockscout#2636) we started fetching transactions and rewards in parallel using elixir tasks. This PR uses similar approach to fetch validation and transaction counters on address page. * Add caches updating independently from Indexer * replace all transaction and validation count methods * fix build * add CHANGELOG entry * Created new layout for mobile menu * Modified navbar.scss removed unnecessary lines * fetch token counters in parallel the slowest part of the token overview page is fetching of total token transfers and total token holders counters queries. Sometimes they even cause page load failures. Now we will start fetching these counters in parallel and limiting query execution to 40 seconds. If they fail to load, we won't show these counters. * add CHANGELOG entry * fix gettext * Removed Unnecessary spaces from navbar.scss file * Delete settings.json Deleted untracked file from my code editor. * ESLINT tests should pass now * ESLINT tests should pass now. Added some changes to fix ESLINT issues * Gettext test should pass * ESLINT test should pass, merging conflict should be solved * ESLINT tests failure recheck * ESLINT tests failure recheck1 * ESLINT tests failure recheck2 * ESLINT tests failure recheck3 * Added update to Changelog * New theme for xUSDT created * All requested changes for mobile menu performed * Removed spaces in dark-theme.scss file * Final changes for xUSDT theme performed * add test for all supported chains * fix typo * Logos added * 2.0.4 * bump version: patch 2.0.4 * Added icons for new network, added non-critical value etc * Added changelog * changed icon value to &-xusdt-chain * New xdai theme styles added * Fix for dashboard banner * Added Changelog.md * All requested changes performed. * Update CHANGELOG.md * Added fixed height for logo. Jumping issue should be fixed now. Added correct hex color. * Gettext issue should be fixed. * Added fixed height for card chain blocks and transactions. * Added CHANGELOG.md entry * Added dai_logo.svg * Use Task.start instead of Task.async in caches * Revert "Filter pending logs" * Created new loaders for blocks chain container, added 2 more loaders to transaction card, fixed height issue. * add CHANGELOG entry * Internationalization added * Resolve merging conflict. * Gettext issue fixed. * Update CHANGELOG.md * Update CHANGELOG.md * remove nonconsensus token transfers * Update CHANGELOG.md * Update CHANGELOG.md * remove nonconsensus logs * use existing forked transactions query * Added new function for new-button * use query across all inserted blocks * fix try it out section * fix CR issues * add CHANGELOG entry * add CHANGELOG entry * add block_number index * remove transaction index creation * deduplicate numbers * Update blocks.ex * fix exchange rate websocket update for Rootstock Rootstock has custom logic for market cap calculation that uses data from DB. This PR add required feilds to exchange_rate when sending it through a web socket. * add CHANGELOG entry * do not update chart * remove nonconsensus internal transactions * remove update of block number for internal transactions * add CHANGELOG entry * fix typo * do not update fetched_coin_balance with nil Currently, we update `fetched_coin_balance` field of an address to nil if changes have `fetched_coin_balance_block_number` bigger than current value in the DB. * add CHANGELOG entry * Update blocks.ex * Exclude nonconsensus blocks from avg block time calculation by default * Enforce DB transaction's order to prevent deadlocks Problem: due to the ShareLock mechanism used by PostgreSQL when there are several DB transactions are acting on multiple rows of the same table, it's possible to incur in a deadlock and so into an error. Solution: enforce a consistent order of ShareLock acquisition in all the relevant transaction. Also add clear documentation that should help in the future. * Enforce DB transaction's order between tables to prevent deadlocks * Update CHANGELOG * fix empty total_supply in coin gecko response * add CHANGELOG entry * canvas * Updated Changelog.md * Add ETS-based cache for accounts page * add log index to logs view * add CHANGELOG entry * add comment * Fix stuck label and value for uncle block height * Improve speed of nonconsensus data removal Problem: removal of nonconsensus data is too inefficient and as a result blocks are imported too slow. Solution: reformulation of deletion logic for better performance * changelog entry * get rid of json schema warnings * add CHANGELOG entry * fix ci * bump cache version * add CHANGELOG entry * Addresses counter
jcortejoso
added a commit
to celo-org/blockscout
that referenced
this pull request
Oct 17, 2019
* update env-vars doc file * parse decimal data from genesis chain spec * svg images for network images * remove CoinMarketCap logic * fix test * add new column to env vars * mock actual request * Add scheme in api base url * Add host in an example of curl request * fix test * fix env var description * checkout package-lock.json from origin/master * Re-added internationalization, re-checked all changes on all devices * Re-added internationalization, re-checked all changes on all devices, resolved merge conflict * Add scheme in api urls * Move api's url to the end of example * Use pipeline * Added upstreamrepo for local repo,fetched updates, re-added internalization * Added Xerom to list of Additional Chains using BlockScout. * Added PR * Fixed incorrect numerical order in Chores * Fetched last updates from forked repo * Changelog.md Clean up * README.md Clean up * Update README.md * Fixed tooltips issue, header should work fine, fixed buttons colors on Contract Address Details page * Changes staging * Changed text color on top miner address tooltip * Set not full path to api on ajax queries * Tried to resolve gettext issue again * Removed unnecessary lines. * Delete _erc721_token_image_container.scss * Update _dashboard-banner.scss * Update overview.html.eex * Update _details.html.eex * Update index.html.eex * Gettext issue should be fixed * Fixed buttons color at smart contract section * Update README.md LUKSO L14 testnet in the list of hosted chains * fix market_cap calculation * fix tests * add fallback * check if blockscout comment is already added * update gettext * add default case * add CHANGELOG entry * fix gettext * fix tests * fix remaining tests * Revert some unintentional changes * Additional cleanup * gettext fix * Cleanup _dashboard-banner.scss * Add CHANGELOG entry * Remove unnecessary whitespaces * set correct last calue for coin balances chart data Sometimes the last value for coin history chart data is incorrect because when selecting records they are grouped by day and max value is selected in that day. So if there are a couple of coin balance changes only max value is shown. This PR always shows the latest value from the database by block number. * add CHANGELOG entry * fix test * fix tests * fetch coin gecko id based on coin symbol Coin Gecko provides API with all listed coins. We can use this API to find coin id instead of setting it by hand. * add CHANGELOG entry * remove config entry * find index of the comment instead of item * fetch address counters async In this PR (blockscout#2636) we started fetching transactions and rewards in parallel using elixir tasks. This PR uses similar approach to fetch validation and transaction counters on address page. * Add caches updating independently from Indexer * replace all transaction and validation count methods * fix build * add CHANGELOG entry * Created new layout for mobile menu * Modified navbar.scss removed unnecessary lines * fetch token counters in parallel the slowest part of the token overview page is fetching of total token transfers and total token holders counters queries. Sometimes they even cause page load failures. Now we will start fetching these counters in parallel and limiting query execution to 40 seconds. If they fail to load, we won't show these counters. * add CHANGELOG entry * fix gettext * Removed Unnecessary spaces from navbar.scss file * Delete settings.json Deleted untracked file from my code editor. * ESLINT tests should pass now * ESLINT tests should pass now. Added some changes to fix ESLINT issues * Gettext test should pass * ESLINT test should pass, merging conflict should be solved * ESLINT tests failure recheck * ESLINT tests failure recheck1 * ESLINT tests failure recheck2 * ESLINT tests failure recheck3 * Added update to Changelog * New theme for xUSDT created * All requested changes for mobile menu performed * Removed spaces in dark-theme.scss file * Final changes for xUSDT theme performed * add test for all supported chains * fix typo * Logos added * 2.0.4 * bump version: patch 2.0.4 * Added icons for new network, added non-critical value etc * Added changelog * changed icon value to &-xusdt-chain * New xdai theme styles added * Fix for dashboard banner * Added Changelog.md * All requested changes performed. * Update CHANGELOG.md * Added fixed height for logo. Jumping issue should be fixed now. Added correct hex color. * Gettext issue should be fixed. * Added fixed height for card chain blocks and transactions. * Added CHANGELOG.md entry * Added dai_logo.svg * Use Task.start instead of Task.async in caches * Revert "Filter pending logs" * Created new loaders for blocks chain container, added 2 more loaders to transaction card, fixed height issue. * add CHANGELOG entry * Internationalization added * Resolve merging conflict. * Gettext issue fixed. * Update CHANGELOG.md * Update CHANGELOG.md * remove nonconsensus token transfers * Update CHANGELOG.md * Update CHANGELOG.md * remove nonconsensus logs * use existing forked transactions query * Added new function for new-button * use query across all inserted blocks * fix CR issues * add CHANGELOG entry * add block_number index * remove transaction index creation * deduplicate numbers * Update blocks.ex * fix exchange rate websocket update for Rootstock Rootstock has custom logic for market cap calculation that uses data from DB. This PR add required feilds to exchange_rate when sending it through a web socket. * add CHANGELOG entry * do not update chart * remove nonconsensus internal transactions * remove update of block number for internal transactions * add CHANGELOG entry * fix typo * do not update fetched_coin_balance with nil Currently, we update `fetched_coin_balance` field of an address to nil if changes have `fetched_coin_balance_block_number` bigger than current value in the DB. * add CHANGELOG entry * Update blocks.ex * Exclude nonconsensus blocks from avg block time calculation by default * Enforce DB transaction's order to prevent deadlocks Problem: due to the ShareLock mechanism used by PostgreSQL when there are several DB transactions are acting on multiple rows of the same table, it's possible to incur in a deadlock and so into an error. Solution: enforce a consistent order of ShareLock acquisition in all the relevant transaction. Also add clear documentation that should help in the future. * Enforce DB transaction's order between tables to prevent deadlocks * Update CHANGELOG * fix empty total_supply in coin gecko response * add CHANGELOG entry * canvas * Updated Changelog.md * Add ETS-based cache for accounts page * add comment * Fix stuck label and value for uncle block height * Added feeRecipient field * link to feeRecipient * Added some changes from repo fork * Changes from Celo's fork * We do not need two images * Changed labels Fee Recipient and cGLD * Fix problem with Postgrex listener database connection * Removed unneeded files
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of #2306
Changelog
Enhancements
The query used to fetch transactions for an address have been re-split and parallelized.
Note that the reason this was done is that, for some addresses more than others, executing a single query on the Transactions table with several
whereclauses is a lot slower than making them separately.This may seem to contradict a previous change where the opposite operation has been done, but the difference is in the fact that previously the list of queries was run sequentially.
Checklist for your PR
CHANGELOG.mdwith this PRmasterin theVersioncolumn.