Metrics: Disk IO reads and writes #890
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… miguel/disk-io-metrics
darrenvechain
approved these changes
Nov 22, 2024
libotony
pushed a commit
that referenced
this pull request
Nov 29, 2024
* changes * removed log * sleeping any way * pr review
otherview
added a commit
that referenced
this pull request
Nov 29, 2024
* Thor client (#818) * feat: add thorclient * refactor: remove roundTripper * refactor: change null check * clean: remove commented code * feat: add account revision and pending tx * fix: add licence headers and fix linter issue * refactor: rename package * refactor: change revision type to string * refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers * refactor: change FilterEvents and FilterTransactions request type to EventFilter * Adding common.EventWrapper to handle channel errors * tweak * update rawclient + update account tests * tidy up names * update tests * pr comments * adding raw tx * Tidy up method names and calls * options client * tweaks * pr comments * Update thorclient/common/common.go Co-authored-by: libotony <[email protected]> * pr comments * Adding Subscriptions * Pr comments * adjust func orders * pr comments * changing subscribe to use the channel close vs multiple channels * adding go-doc * no error after unsubscribe * pr comments * checking status code is 2xx * fix: change FilterTransfers argument --------- Co-authored-by: otherview <[email protected]> Co-authored-by: libotony <[email protected]> * Add testchain package (#844) * Refactor thor node * thorchain allows insertion of blocks * remove thorNode, added testchain * clean up + comments * adding license headers * adding templating tests for thorclient * Remove test event hacks * remove types * removed chain_builder + added logdb to testchain * pr comments * Update test/testchain/chain.go Co-authored-by: libotony <[email protected]> --------- Co-authored-by: libotony <[email protected]> * cmd/thor: update instance dir to v4 * trie: implement varint-prefix coder * deps: add github.com/qianbin/drlp * trie: implement appendHexToCompact & compactLen * trie: temporarily remove merkle proof stuff * trie: many changes * disk usage reduced by 33% (force embedding shortnode) * new encoding method for storing nodes * optimize trie hashing * versioning standalone nodes * remove extended trie * improve trie interface * simplify NodeIterator, remove unused codes * trie: optimize full-node encoding/decoding * trie: tweak shortnode encoding * muxdb: move engine pkg * trie: add Version() method for node interface * muxdb: refactor due to trie updates and: * remove leafbank stuff * simplify muxdb.Trie implementation * improve root node cache using ttl eviction * add leaf key filter * chain: a lot of changes * improve block content storage scheme * remove steady block tracking * remove tx & receipt cache * state: changes due to update of trie * lowrlp: remove this pkg * txpool: changes due to underlying pkg update * genesis: changes due to underlying pkg update * consensus: changes due to underlying pkg update * builtin: changes due to underlying pkg update * runtime: changes due to underlying pkg update * api: changes due to underlying pkg update * cmd/thor/pruner: rename pkg optimizer to pruner * cmd/thor: changes due to underlying pkg update * muxdb: abandon leaf filter * cmd/thor/pruner: use smaller period when nearly synced * muxdb: improve trie node path encoding * trie: treat short nodes as standalone nodes when skipping hash * cmd/thor: fix disablePrunerFlag not work * trie: improve refNode encoding/decoding * muxdb: improve history node key encoding * cmd/thor: adjust pruner parameters * build: fix test cases * lint: fix lint error * muxdb: fix ver encoding in node blob cache * muxdb: add test cases for cache * runtime: fix test compile error * make build and test pass after rebase * add back pruner tests * add tests for node encoding * minor typo * update named store space prefix * add more tests * fix block summary in repo * make build and test pass after rebase * add back pruner tests * remove SetBestBlockID from tests * minor improvement * pr comments * adding a comment * Metrics: Cache hit/miss (#886) * change * reverted to previous format * Add dummy cache for inmem ops (#883) * Add empty cache for inmem ops * changing name to dummyCache * Metrics: Reuse `shouldLog` so we get cache hit/miss data at the same pace (#888) * first commit * first commit * Metrics: Disk IO reads and writes (#890) * changes * removed log * sleeping any way * pr review * Pedro/maindb v4/benchmarks (#891) * Adding Benchmark tests * processing txs * Working benchmarks * lint * adding tempdir * Adding transactions benchmark + repository cache * improve cache stats log and metric * totally removed SetBestBlockID * removed unused tests * update bench tests * getreceipts metrics + lint --------- Co-authored-by: Paolo Galli <[email protected]> Co-authored-by: libotony <[email protected]> Co-authored-by: qianbin <[email protected]> Co-authored-by: Darren Kelly <[email protected]> Co-authored-by: Miguel Angel Rojo <[email protected]>
darrenvechain
added a commit
that referenced
this pull request
Dec 3, 2024
* Thor client (#818) * feat: add thorclient * refactor: remove roundTripper * refactor: change null check * clean: remove commented code * feat: add account revision and pending tx * fix: add licence headers and fix linter issue * refactor: rename package * refactor: change revision type to string * refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers * refactor: change FilterEvents and FilterTransactions request type to EventFilter * Adding common.EventWrapper to handle channel errors * tweak * update rawclient + update account tests * tidy up names * update tests * pr comments * adding raw tx * Tidy up method names and calls * options client * tweaks * pr comments * Update thorclient/common/common.go Co-authored-by: libotony <[email protected]> * pr comments * Adding Subscriptions * Pr comments * adjust func orders * pr comments * changing subscribe to use the channel close vs multiple channels * adding go-doc * no error after unsubscribe * pr comments * checking status code is 2xx * fix: change FilterTransfers argument --------- Co-authored-by: otherview <[email protected]> Co-authored-by: libotony <[email protected]> * Add testchain package (#844) * Refactor thor node * thorchain allows insertion of blocks * remove thorNode, added testchain * clean up + comments * adding license headers * adding templating tests for thorclient * Remove test event hacks * remove types * removed chain_builder + added logdb to testchain * pr comments * Update test/testchain/chain.go Co-authored-by: libotony <[email protected]> --------- Co-authored-by: libotony <[email protected]> * cmd/thor: update instance dir to v4 * trie: implement varint-prefix coder * deps: add github.com/qianbin/drlp * trie: implement appendHexToCompact & compactLen * trie: temporarily remove merkle proof stuff * trie: many changes * disk usage reduced by 33% (force embedding shortnode) * new encoding method for storing nodes * optimize trie hashing * versioning standalone nodes * remove extended trie * improve trie interface * simplify NodeIterator, remove unused codes * trie: optimize full-node encoding/decoding * trie: tweak shortnode encoding * muxdb: move engine pkg * trie: add Version() method for node interface * muxdb: refactor due to trie updates and: * remove leafbank stuff * simplify muxdb.Trie implementation * improve root node cache using ttl eviction * add leaf key filter * chain: a lot of changes * improve block content storage scheme * remove steady block tracking * remove tx & receipt cache * state: changes due to update of trie * lowrlp: remove this pkg * txpool: changes due to underlying pkg update * genesis: changes due to underlying pkg update * consensus: changes due to underlying pkg update * builtin: changes due to underlying pkg update * runtime: changes due to underlying pkg update * api: changes due to underlying pkg update * cmd/thor/pruner: rename pkg optimizer to pruner * cmd/thor: changes due to underlying pkg update * muxdb: abandon leaf filter * cmd/thor/pruner: use smaller period when nearly synced * muxdb: improve trie node path encoding * trie: treat short nodes as standalone nodes when skipping hash * cmd/thor: fix disablePrunerFlag not work * trie: improve refNode encoding/decoding * muxdb: improve history node key encoding * cmd/thor: adjust pruner parameters * build: fix test cases * lint: fix lint error * muxdb: fix ver encoding in node blob cache * muxdb: add test cases for cache * runtime: fix test compile error * make build and test pass after rebase * add back pruner tests * add tests for node encoding * minor typo * update named store space prefix * add more tests * fix block summary in repo * make build and test pass after rebase * add back pruner tests * remove SetBestBlockID from tests * minor improvement * pr comments * adding a comment * Metrics: Cache hit/miss (#886) * change * reverted to previous format * Add dummy cache for inmem ops (#883) * Add empty cache for inmem ops * changing name to dummyCache * Metrics: Reuse `shouldLog` so we get cache hit/miss data at the same pace (#888) * first commit * first commit * Metrics: Disk IO reads and writes (#890) * changes * removed log * sleeping any way * pr review * Pedro/maindb v4/benchmarks (#891) * Adding Benchmark tests * processing txs * Working benchmarks * lint * adding tempdir * improve cache stats log and metric * totally removed SetBestBlockID * fix: logs API not returning results when to=0,from=omitted * minor update * fix: PR comments + lint * improve convert range logic * chore: remove debug log --------- Co-authored-by: Paolo Galli <[email protected]> Co-authored-by: otherview <[email protected]> Co-authored-by: libotony <[email protected]> Co-authored-by: qianbin <[email protected]> Co-authored-by: Miguel Angel Rojo <[email protected]>
darrenvechain
added a commit
that referenced
this pull request
Dec 9, 2024
* fix(documentation): use absolute links in markdown (#889) * Add benchmark test to node block process (#892) * Add benchmark test to node block process * added file-based storage * use tempdir * update dependency go-ethereum (#895) * chore: update API metrics bucket and endpoint names (#893) * chore: update API metrics bucket and endpoint names * fix: typo & tests * fix: lint * chore: add websocket total counter * fix: txs endpoints names & ws subject * fix: unit tests * chore: standardise naming convention * chore: add websocke duration & http code * chore: add websocke duration & http code * fix: lint issues * fix: sync issues with metrics * chore: update websocket durations bucket * fix: PR comments - use sync.Once * chore: update builtin generation (#896) * chore: update builtin generation * fix: update GHA * Thor client (#818) * feat: add thorclient * refactor: remove roundTripper * refactor: change null check * clean: remove commented code * feat: add account revision and pending tx * fix: add licence headers and fix linter issue * refactor: rename package * refactor: change revision type to string * refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers * refactor: change FilterEvents and FilterTransactions request type to EventFilter * Adding common.EventWrapper to handle channel errors * tweak * update rawclient + update account tests * tidy up names * update tests * pr comments * adding raw tx * Tidy up method names and calls * options client * tweaks * pr comments * Update thorclient/common/common.go Co-authored-by: libotony <[email protected]> * pr comments * Adding Subscriptions * Pr comments * adjust func orders * pr comments * changing subscribe to use the channel close vs multiple channels * adding go-doc * no error after unsubscribe * pr comments * checking status code is 2xx * fix: change FilterTransfers argument --------- Co-authored-by: otherview <[email protected]> Co-authored-by: libotony <[email protected]> * Add testchain package (#844) * Refactor thor node * thorchain allows insertion of blocks * remove thorNode, added testchain * clean up + comments * adding license headers * adding templating tests for thorclient * Remove test event hacks * remove types * removed chain_builder + added logdb to testchain * pr comments * Update test/testchain/chain.go Co-authored-by: libotony <[email protected]> --------- Co-authored-by: libotony <[email protected]> * cmd/thor: update instance dir to v4 * trie: implement varint-prefix coder * deps: add github.com/qianbin/drlp * trie: implement appendHexToCompact & compactLen * trie: temporarily remove merkle proof stuff * trie: many changes * disk usage reduced by 33% (force embedding shortnode) * new encoding method for storing nodes * optimize trie hashing * versioning standalone nodes * remove extended trie * improve trie interface * simplify NodeIterator, remove unused codes * trie: optimize full-node encoding/decoding * trie: tweak shortnode encoding * muxdb: move engine pkg * trie: add Version() method for node interface * muxdb: refactor due to trie updates and: * remove leafbank stuff * simplify muxdb.Trie implementation * improve root node cache using ttl eviction * add leaf key filter * chain: a lot of changes * improve block content storage scheme * remove steady block tracking * remove tx & receipt cache * state: changes due to update of trie * lowrlp: remove this pkg * txpool: changes due to underlying pkg update * genesis: changes due to underlying pkg update * consensus: changes due to underlying pkg update * builtin: changes due to underlying pkg update * runtime: changes due to underlying pkg update * api: changes due to underlying pkg update * cmd/thor/pruner: rename pkg optimizer to pruner * cmd/thor: changes due to underlying pkg update * muxdb: abandon leaf filter * cmd/thor/pruner: use smaller period when nearly synced * muxdb: improve trie node path encoding * trie: treat short nodes as standalone nodes when skipping hash * cmd/thor: fix disablePrunerFlag not work * trie: improve refNode encoding/decoding * muxdb: improve history node key encoding * cmd/thor: adjust pruner parameters * build: fix test cases * lint: fix lint error * muxdb: fix ver encoding in node blob cache * muxdb: add test cases for cache * runtime: fix test compile error * make build and test pass after rebase * add back pruner tests * add tests for node encoding * minor typo * update named store space prefix * add more tests * fix block summary in repo * make build and test pass after rebase * add back pruner tests * remove SetBestBlockID from tests * minor improvement * pr comments * adding a comment * Metrics: Cache hit/miss (#886) * change * reverted to previous format * Add dummy cache for inmem ops (#883) * Add empty cache for inmem ops * changing name to dummyCache * Metrics: Reuse `shouldLog` so we get cache hit/miss data at the same pace (#888) * first commit * first commit * Metrics: Disk IO reads and writes (#890) * changes * removed log * sleeping any way * pr review * Pedro/maindb v4/benchmarks (#891) * Adding Benchmark tests * processing txs * Working benchmarks * lint * adding tempdir * improve cache stats log and metric * totally removed SetBestBlockID * Maindb v4 Transaction benchmark plus cache (#894) * Thor client (#818) * feat: add thorclient * refactor: remove roundTripper * refactor: change null check * clean: remove commented code * feat: add account revision and pending tx * fix: add licence headers and fix linter issue * refactor: rename package * refactor: change revision type to string * refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers * refactor: change FilterEvents and FilterTransactions request type to EventFilter * Adding common.EventWrapper to handle channel errors * tweak * update rawclient + update account tests * tidy up names * update tests * pr comments * adding raw tx * Tidy up method names and calls * options client * tweaks * pr comments * Update thorclient/common/common.go Co-authored-by: libotony <[email protected]> * pr comments * Adding Subscriptions * Pr comments * adjust func orders * pr comments * changing subscribe to use the channel close vs multiple channels * adding go-doc * no error after unsubscribe * pr comments * checking status code is 2xx * fix: change FilterTransfers argument --------- Co-authored-by: otherview <[email protected]> Co-authored-by: libotony <[email protected]> * Add testchain package (#844) * Refactor thor node * thorchain allows insertion of blocks * remove thorNode, added testchain * clean up + comments * adding license headers * adding templating tests for thorclient * Remove test event hacks * remove types * removed chain_builder + added logdb to testchain * pr comments * Update test/testchain/chain.go Co-authored-by: libotony <[email protected]> --------- Co-authored-by: libotony <[email protected]> * cmd/thor: update instance dir to v4 * trie: implement varint-prefix coder * deps: add github.com/qianbin/drlp * trie: implement appendHexToCompact & compactLen * trie: temporarily remove merkle proof stuff * trie: many changes * disk usage reduced by 33% (force embedding shortnode) * new encoding method for storing nodes * optimize trie hashing * versioning standalone nodes * remove extended trie * improve trie interface * simplify NodeIterator, remove unused codes * trie: optimize full-node encoding/decoding * trie: tweak shortnode encoding * muxdb: move engine pkg * trie: add Version() method for node interface * muxdb: refactor due to trie updates and: * remove leafbank stuff * simplify muxdb.Trie implementation * improve root node cache using ttl eviction * add leaf key filter * chain: a lot of changes * improve block content storage scheme * remove steady block tracking * remove tx & receipt cache * state: changes due to update of trie * lowrlp: remove this pkg * txpool: changes due to underlying pkg update * genesis: changes due to underlying pkg update * consensus: changes due to underlying pkg update * builtin: changes due to underlying pkg update * runtime: changes due to underlying pkg update * api: changes due to underlying pkg update * cmd/thor/pruner: rename pkg optimizer to pruner * cmd/thor: changes due to underlying pkg update * muxdb: abandon leaf filter * cmd/thor/pruner: use smaller period when nearly synced * muxdb: improve trie node path encoding * trie: treat short nodes as standalone nodes when skipping hash * cmd/thor: fix disablePrunerFlag not work * trie: improve refNode encoding/decoding * muxdb: improve history node key encoding * cmd/thor: adjust pruner parameters * build: fix test cases * lint: fix lint error * muxdb: fix ver encoding in node blob cache * muxdb: add test cases for cache * runtime: fix test compile error * make build and test pass after rebase * add back pruner tests * add tests for node encoding * minor typo * update named store space prefix * add more tests * fix block summary in repo * make build and test pass after rebase * add back pruner tests * remove SetBestBlockID from tests * minor improvement * pr comments * adding a comment * Metrics: Cache hit/miss (#886) * change * reverted to previous format * Add dummy cache for inmem ops (#883) * Add empty cache for inmem ops * changing name to dummyCache * Metrics: Reuse `shouldLog` so we get cache hit/miss data at the same pace (#888) * first commit * first commit * Metrics: Disk IO reads and writes (#890) * changes * removed log * sleeping any way * pr review * Pedro/maindb v4/benchmarks (#891) * Adding Benchmark tests * processing txs * Working benchmarks * lint * adding tempdir * Adding transactions benchmark + repository cache * improve cache stats log and metric * totally removed SetBestBlockID * removed unused tests * update bench tests * getreceipts metrics + lint --------- Co-authored-by: Paolo Galli <[email protected]> Co-authored-by: libotony <[email protected]> Co-authored-by: qianbin <[email protected]> Co-authored-by: Darren Kelly <[email protected]> Co-authored-by: Miguel Angel Rojo <[email protected]> * reduce clauses() allocations * bug: fix logs endpoints query (#900) * Thor client (#818) * feat: add thorclient * refactor: remove roundTripper * refactor: change null check * clean: remove commented code * feat: add account revision and pending tx * fix: add licence headers and fix linter issue * refactor: rename package * refactor: change revision type to string * refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers * refactor: change FilterEvents and FilterTransactions request type to EventFilter * Adding common.EventWrapper to handle channel errors * tweak * update rawclient + update account tests * tidy up names * update tests * pr comments * adding raw tx * Tidy up method names and calls * options client * tweaks * pr comments * Update thorclient/common/common.go Co-authored-by: libotony <[email protected]> * pr comments * Adding Subscriptions * Pr comments * adjust func orders * pr comments * changing subscribe to use the channel close vs multiple channels * adding go-doc * no error after unsubscribe * pr comments * checking status code is 2xx * fix: change FilterTransfers argument --------- Co-authored-by: otherview <[email protected]> Co-authored-by: libotony <[email protected]> * Add testchain package (#844) * Refactor thor node * thorchain allows insertion of blocks * remove thorNode, added testchain * clean up + comments * adding license headers * adding templating tests for thorclient * Remove test event hacks * remove types * removed chain_builder + added logdb to testchain * pr comments * Update test/testchain/chain.go Co-authored-by: libotony <[email protected]> --------- Co-authored-by: libotony <[email protected]> * cmd/thor: update instance dir to v4 * trie: implement varint-prefix coder * deps: add github.com/qianbin/drlp * trie: implement appendHexToCompact & compactLen * trie: temporarily remove merkle proof stuff * trie: many changes * disk usage reduced by 33% (force embedding shortnode) * new encoding method for storing nodes * optimize trie hashing * versioning standalone nodes * remove extended trie * improve trie interface * simplify NodeIterator, remove unused codes * trie: optimize full-node encoding/decoding * trie: tweak shortnode encoding * muxdb: move engine pkg * trie: add Version() method for node interface * muxdb: refactor due to trie updates and: * remove leafbank stuff * simplify muxdb.Trie implementation * improve root node cache using ttl eviction * add leaf key filter * chain: a lot of changes * improve block content storage scheme * remove steady block tracking * remove tx & receipt cache * state: changes due to update of trie * lowrlp: remove this pkg * txpool: changes due to underlying pkg update * genesis: changes due to underlying pkg update * consensus: changes due to underlying pkg update * builtin: changes due to underlying pkg update * runtime: changes due to underlying pkg update * api: changes due to underlying pkg update * cmd/thor/pruner: rename pkg optimizer to pruner * cmd/thor: changes due to underlying pkg update * muxdb: abandon leaf filter * cmd/thor/pruner: use smaller period when nearly synced * muxdb: improve trie node path encoding * trie: treat short nodes as standalone nodes when skipping hash * cmd/thor: fix disablePrunerFlag not work * trie: improve refNode encoding/decoding * muxdb: improve history node key encoding * cmd/thor: adjust pruner parameters * build: fix test cases * lint: fix lint error * muxdb: fix ver encoding in node blob cache * muxdb: add test cases for cache * runtime: fix test compile error * make build and test pass after rebase * add back pruner tests * add tests for node encoding * minor typo * update named store space prefix * add more tests * fix block summary in repo * make build and test pass after rebase * add back pruner tests * remove SetBestBlockID from tests * minor improvement * pr comments * adding a comment * Metrics: Cache hit/miss (#886) * change * reverted to previous format * Add dummy cache for inmem ops (#883) * Add empty cache for inmem ops * changing name to dummyCache * Metrics: Reuse `shouldLog` so we get cache hit/miss data at the same pace (#888) * first commit * first commit * Metrics: Disk IO reads and writes (#890) * changes * removed log * sleeping any way * pr review * Pedro/maindb v4/benchmarks (#891) * Adding Benchmark tests * processing txs * Working benchmarks * lint * adding tempdir * improve cache stats log and metric * totally removed SetBestBlockID * fix: logs API not returning results when to=0,from=omitted * minor update * fix: PR comments + lint * improve convert range logic * chore: remove debug log --------- Co-authored-by: Paolo Galli <[email protected]> Co-authored-by: otherview <[email protected]> Co-authored-by: libotony <[email protected]> Co-authored-by: qianbin <[email protected]> Co-authored-by: Miguel Angel Rojo <[email protected]> * chore(chain): add repo cache metrics (#910) * chore(chain): add repo cache metrics * refactor(chain): cache hit miss --------- Co-authored-by: Darren Kelly <[email protected]> Co-authored-by: Pedro Gomes <[email protected]> Co-authored-by: Paolo Galli <[email protected]> Co-authored-by: qianbin <[email protected]> Co-authored-by: Miguel Angel Rojo <[email protected]> Co-authored-by: Darren Kelly <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Collecting Disk I/O data (reads and writes only, not bytes for now) every 5 seconds. In the following graph we are applying
rateas usual (please suggest any other representation).If you want to reproduce it on your side, run it with Docker (the code assumes that your are running
thorin Linux - I have a macOS - , use your localdatafolder):Some logs (not part of the current code, follow
LLEGA):