Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add testchain package #844

Merged
merged 16 commits into from
Nov 7, 2024
Merged

Add testchain package #844

merged 16 commits into from
Nov 7, 2024

Conversation

otherview
Copy link
Member

@otherview otherview commented Sep 10, 2024

Description

  • Added testchain package
    testchain package is meant to be used for testing.
    It simplifies the common uses for the thor chain in tests.
    Wraps most of the most used operations like mint a transaction.
    It has a chain builder that allows to create thor chain with a different subset of components (inmem vs realdb, etc..)

  • Added eventcontract package
    eventcontract package is a precompiled contract
    It's meant to help trigger events on-chain and test these ( subscriptions and events tests )

  • Simplified the initChain logic
    initChain now returns a testchain and can be easily manipulated in-test

  • Removed (many but not all) global test vars

  • Added thorclient/api_test.go which allows to test requests and response payloads for several apis

Fixes # (issue)

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also
list any relevant details for your test configuration

  • Test A
  • Test B

Test Configuration:

  • Go Version:
  • Hardware:
  • Docker Version:

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • New and existing E2E tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules
  • I have not added any vulnerable dependencies to my code

@otherview otherview force-pushed the pedro/refactor_thor_node branch 2 times, most recently from 5ed1c53 to 4ded72c Compare September 11, 2024 16:54
@otherview otherview force-pushed the pedro/refactor_thor_node branch from 4ded72c to c41489d Compare September 11, 2024 17:08
@codecov-commenter
Copy link

codecov-commenter commented Sep 12, 2024

Codecov Report

Attention: Patch coverage is 70.68966% with 34 lines in your changes missing coverage. Please review.

Project coverage is 60.62%. Comparing base (8be8574) to head (e84d59c).

Files with missing lines Patch % Lines
test/testchain/chain.go 71.30% 22 Missing and 11 partials ⚠️
cmd/thor/solo/types.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #844      +/-   ##
==========================================
+ Coverage   60.55%   60.62%   +0.06%     
==========================================
  Files         214      215       +1     
  Lines       22984    23099     +115     
==========================================
+ Hits        13918    14003      +85     
- Misses       7925     7947      +22     
- Partials     1141     1149       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@otherview otherview changed the title Refactor thor node Add testchain package Sep 16, 2024
@otherview otherview force-pushed the pedro/refactor_thor_node branch from 16868fc to c5c8423 Compare September 16, 2024 10:00
@otherview otherview marked this pull request as ready for review September 16, 2024 13:41
@otherview otherview requested a review from a team as a code owner September 16, 2024 13:41
@darrenvechain
Copy link
Member

IMO the test package should be renamed to internal so that is not accessible by clients with thor as a dependency. It would only be available within thor itself, similar to how geth has this:

https://github.com/ethereum/go-ethereum/tree/master/internal

https://dev.to/stefanalfbo/internal-packages-in-go-2h9

@otherview
Copy link
Member Author

IMO the test package should be renamed to internal so that is not accessible by clients with thor as a dependency. It would only be available within thor itself, similar to how geth has this:

https://github.com/ethereum/go-ethereum/tree/master/internal

https://dev.to/stefanalfbo/internal-packages-in-go-2h9

No strong feelings either way. But would avoid doing this until it's a necessity.

@libotony
Copy link
Member

IMO the test package should be renamed to internal so that is not accessible by clients with thor as a dependency. It would only be available within thor itself, similar to how geth has this:
https://github.com/ethereum/go-ethereum/tree/master/internal
https://dev.to/stefanalfbo/internal-packages-in-go-2h9

No strong feelings either way. But would avoid doing this until it's a necessity.

Looks like test-oriented to me.

@darrenvechain
Copy link
Member

darrenvechain commented Nov 5, 2024

No strong feelings either way. But would avoid doing this until it's a necessity.

IMO its not a matter of necessity, its just cleaner / standard practice

test/testchain/chain.go Outdated Show resolved Hide resolved
test/testchain/chain.go Outdated Show resolved Hide resolved
test/testchain/chain.go Outdated Show resolved Hide resolved
libotony
libotony previously approved these changes Nov 7, 2024
@otherview otherview merged commit 784604d into master Nov 7, 2024
15 checks passed
@otherview otherview deleted the pedro/refactor_thor_node branch November 7, 2024 10:57
paologalligit added a commit that referenced this pull request Nov 8, 2024
* Thor client (#818)

* feat: add thorclient

* refactor: remove roundTripper

* refactor: change null check

* clean: remove commented code

* feat: add account revision and pending tx

* fix: add licence headers and fix linter issue

* refactor: rename package

* refactor: change revision type to string

* refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers

* refactor: change FilterEvents and FilterTransactions request type to EventFilter

* Adding common.EventWrapper to handle channel errors

* tweak

* update rawclient + update account tests

* tidy up names

* update tests

* pr comments

* adding raw tx

* Tidy up method names and calls

* options client

* tweaks

* pr comments

* Update thorclient/common/common.go

Co-authored-by: libotony <[email protected]>

* pr comments

* Adding Subscriptions

* Pr comments

* adjust func orders

* pr comments

* changing subscribe to use the channel close vs multiple channels

* adding go-doc

* no error after unsubscribe

* pr comments

* checking status code is 2xx

* fix: change FilterTransfers argument

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>

* Show all issues on lint (#869)

* Show all issues on lint

* fix lint

* fix(docker): using AWS docker repo for trivy (#872)

* fix(docker): using AWS docker repo for trivy

* fix(docker): using AWS docker repo for trivy

* Darren/feat/add subscription cache (#866)

* ehancement: create a cache for block based subscriptions

* minor: change function names for subscriptions

* test: add unit test for message cache

* chore: add license headers

* refactor: fix up error handling

* fix: remove bad test

* fix: PR comments

* fix: PR comments - remove block cache

* refactor(subscriptions): store structs in cache, not bytes

* fix(license): add license header

* chore(subscriptions): revert unit test changes

* enhancement: resolve pr comments to use simplelru

* enhancement: resolve pr comments - use id as key

* Add additional block tests (#863)

* enhancement(logging): leverage trace level (#873)

* Add testchain package (#844)

* Refactor thor node

* thorchain allows insertion of blocks

* remove thorNode, added testchain

* clean up + comments

* adding license headers

* adding templating tests for thorclient

* Remove test event hacks

* remove types

* removed chain_builder + added logdb to testchain

* pr comments

* Update test/testchain/chain.go

Co-authored-by: libotony <[email protected]>

---------

Co-authored-by: libotony <[email protected]>

* chore(docs): update spec for validator nodes (#875)

* chore(docs): update spec for validator nodes

* chore(docs): update cores

* chore(docs): remove public node stuff

* Darren/logdb remove leading zeros (#865)

* feat: add new txIndex column to event meta response

* test: add convert event test

* feat: make txLog and txIndex as optional return params

* chore: update swagger with new event optional data

* feat: save logIndex in sequence

* feat: tweaked bits in sequence

* refactor: rename optional log meta field

* refactor: comments, yaml and txIndex counts

* rebase to master

* fix: remove stale struct

* add txIndex to returned logdb query

* reset to 0 eventCount and transferCount each receipt and write blockId only once

* fix lint

* rephrase logIndex description in yaml file

* refactor: use filter.Option instead of eventFilter.Option

* move includeIndexes to api

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>
Co-authored-by: Darren Kelly <[email protected]>
Co-authored-by: Makis Christou <[email protected]>
libotony added a commit that referenced this pull request Nov 12, 2024
* Thor client (#818)

* feat: add thorclient

* refactor: remove roundTripper

* refactor: change null check

* clean: remove commented code

* feat: add account revision and pending tx

* fix: add licence headers and fix linter issue

* refactor: rename package

* refactor: change revision type to string

* refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers

* refactor: change FilterEvents and FilterTransactions request type to EventFilter

* Adding common.EventWrapper to handle channel errors

* tweak

* update rawclient + update account tests

* tidy up names

* update tests

* pr comments

* adding raw tx

* Tidy up method names and calls

* options client

* tweaks

* pr comments

* Update thorclient/common/common.go

Co-authored-by: libotony <[email protected]>

* pr comments

* Adding Subscriptions

* Pr comments

* adjust func orders

* pr comments

* changing subscribe to use the channel close vs multiple channels

* adding go-doc

* no error after unsubscribe

* pr comments

* checking status code is 2xx

* fix: change FilterTransfers argument

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>

* Show all issues on lint (#869)

* Show all issues on lint

* fix lint

* fix(docker): using AWS docker repo for trivy (#872)

* fix(docker): using AWS docker repo for trivy

* fix(docker): using AWS docker repo for trivy

* Darren/feat/add subscription cache (#866)

* ehancement: create a cache for block based subscriptions

* minor: change function names for subscriptions

* test: add unit test for message cache

* chore: add license headers

* refactor: fix up error handling

* fix: remove bad test

* fix: PR comments

* fix: PR comments - remove block cache

* refactor(subscriptions): store structs in cache, not bytes

* fix(license): add license header

* chore(subscriptions): revert unit test changes

* enhancement: resolve pr comments to use simplelru

* enhancement: resolve pr comments - use id as key

* Add additional block tests (#863)

* enhancement(logging): leverage trace level (#873)

* Add testchain package (#844)

* Refactor thor node

* thorchain allows insertion of blocks

* remove thorNode, added testchain

* clean up + comments

* adding license headers

* adding templating tests for thorclient

* Remove test event hacks

* remove types

* removed chain_builder + added logdb to testchain

* pr comments

* Update test/testchain/chain.go

Co-authored-by: libotony <[email protected]>

---------

Co-authored-by: libotony <[email protected]>

* chore(docs): update spec for validator nodes (#875)

* chore(docs): update spec for validator nodes

* chore(docs): update cores

* chore(docs): remove public node stuff

* Darren/logdb remove leading zeros (#865)

* feat: add new txIndex column to event meta response

* test: add convert event test

* feat: make txLog and txIndex as optional return params

* chore: update swagger with new event optional data

* feat: save logIndex in sequence

* feat: tweaked bits in sequence

* refactor: rename optional log meta field

* refactor: comments, yaml and txIndex counts

* rebase to master

* fix: remove stale struct

* add txIndex to returned logdb query

* reset to 0 eventCount and transferCount each receipt and write blockId only once

* fix lint

* rephrase logIndex description in yaml file

* refactor: use filter.Option instead of eventFilter.Option

* move includeIndexes to api

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>
Co-authored-by: Darren Kelly <[email protected]>
Co-authored-by: Makis Christou <[email protected]>
libotony added a commit that referenced this pull request Nov 12, 2024
* Refactor thor node

* thorchain allows insertion of blocks

* remove thorNode, added testchain

* clean up + comments

* adding license headers

* adding templating tests for thorclient

* Remove test event hacks

* remove types

* removed chain_builder + added logdb to testchain

* pr comments

* Update test/testchain/chain.go

Co-authored-by: libotony <[email protected]>

---------

Co-authored-by: libotony <[email protected]>
libotony added a commit that referenced this pull request Nov 29, 2024
* Refactor thor node

* thorchain allows insertion of blocks

* remove thorNode, added testchain

* clean up + comments

* adding license headers

* adding templating tests for thorclient

* Remove test event hacks

* remove types

* removed chain_builder + added logdb to testchain

* pr comments

* Update test/testchain/chain.go

Co-authored-by: libotony <[email protected]>

---------

Co-authored-by: libotony <[email protected]>
otherview added a commit that referenced this pull request Nov 29, 2024
* Thor client (#818)

* feat: add thorclient

* refactor: remove roundTripper

* refactor: change null check

* clean: remove commented code

* feat: add account revision and pending tx

* fix: add licence headers and fix linter issue

* refactor: rename package

* refactor: change revision type to string

* refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers

* refactor: change FilterEvents and FilterTransactions request type to EventFilter

* Adding common.EventWrapper to handle channel errors

* tweak

* update rawclient + update account tests

* tidy up names

* update tests

* pr comments

* adding raw tx

* Tidy up method names and calls

* options client

* tweaks

* pr comments

* Update thorclient/common/common.go

Co-authored-by: libotony <[email protected]>

* pr comments

* Adding Subscriptions

* Pr comments

* adjust func orders

* pr comments

* changing subscribe to use the channel close vs multiple channels

* adding go-doc

* no error after unsubscribe

* pr comments

* checking status code is 2xx

* fix: change FilterTransfers argument

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>

* Add testchain package (#844)

* Refactor thor node

* thorchain allows insertion of blocks

* remove thorNode, added testchain

* clean up + comments

* adding license headers

* adding templating tests for thorclient

* Remove test event hacks

* remove types

* removed chain_builder + added logdb to testchain

* pr comments

* Update test/testchain/chain.go

Co-authored-by: libotony <[email protected]>

---------

Co-authored-by: libotony <[email protected]>

* cmd/thor: update instance dir to v4

* trie: implement varint-prefix coder

* deps: add github.com/qianbin/drlp

* trie: implement appendHexToCompact & compactLen

* trie: temporarily remove merkle proof stuff

* trie: many changes

* disk usage reduced by 33% (force embedding shortnode)
* new encoding method for storing nodes
* optimize trie hashing
* versioning standalone nodes
* remove extended trie
* improve trie interface
* simplify NodeIterator, remove unused codes

* trie: optimize full-node encoding/decoding

* trie: tweak shortnode encoding

* muxdb: move engine pkg

* trie: add Version() method for node interface

* muxdb: refactor due to trie updates and:

* remove leafbank stuff
* simplify muxdb.Trie implementation
* improve root node cache using ttl eviction
* add leaf key filter

* chain: a lot of changes

* improve block content storage scheme
* remove steady block tracking
* remove tx & receipt cache

* state: changes due to update of trie

* lowrlp: remove this pkg

* txpool: changes due to underlying pkg update

* genesis: changes due to underlying pkg update

* consensus: changes due to underlying pkg update

* builtin: changes due to underlying pkg update

* runtime: changes due to underlying pkg update

* api: changes due to underlying pkg update

* cmd/thor/pruner: rename pkg optimizer to pruner

* cmd/thor: changes due to underlying pkg update

* muxdb: abandon leaf filter

* cmd/thor/pruner: use smaller period when nearly synced

* muxdb: improve trie node path encoding

* trie: treat short nodes as standalone nodes when skipping hash

* cmd/thor: fix disablePrunerFlag not work

* trie: improve refNode encoding/decoding

* muxdb: improve history node key encoding

* cmd/thor: adjust pruner parameters

* build: fix test cases

* lint: fix lint error

* muxdb: fix ver encoding in node blob cache

* muxdb: add test cases for cache

* runtime: fix test compile error

* make build and test pass after rebase

* add back pruner tests

* add tests for node encoding

* minor typo

* update named store space prefix

* add more tests

* fix block summary in repo

* make build and test pass after rebase

* add back pruner tests

* remove SetBestBlockID from tests

* minor improvement

* pr comments

* adding a comment

* Metrics: Cache hit/miss (#886)

* change

* reverted to previous format

* Add dummy cache for inmem ops (#883)

* Add empty cache for inmem ops

* changing name to dummyCache

* Metrics: Reuse `shouldLog` so we get cache hit/miss data at the same pace (#888)

* first commit

* first commit

* Metrics: Disk IO reads and writes (#890)

* changes

* removed log

* sleeping any way

* pr review

* Pedro/maindb v4/benchmarks (#891)

* Adding Benchmark tests

* processing txs

* Working benchmarks

* lint

* adding tempdir

* Adding transactions benchmark + repository cache

* improve cache stats log and metric

* totally removed SetBestBlockID

* removed unused tests

* update bench tests

* getreceipts metrics + lint

---------

Co-authored-by: Paolo Galli <[email protected]>
Co-authored-by: libotony <[email protected]>
Co-authored-by: qianbin <[email protected]>
Co-authored-by: Darren Kelly <[email protected]>
Co-authored-by: Miguel Angel Rojo <[email protected]>
darrenvechain added a commit that referenced this pull request Dec 3, 2024
* Thor client (#818)

* feat: add thorclient

* refactor: remove roundTripper

* refactor: change null check

* clean: remove commented code

* feat: add account revision and pending tx

* fix: add licence headers and fix linter issue

* refactor: rename package

* refactor: change revision type to string

* refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers

* refactor: change FilterEvents and FilterTransactions request type to EventFilter

* Adding common.EventWrapper to handle channel errors

* tweak

* update rawclient + update account tests

* tidy up names

* update tests

* pr comments

* adding raw tx

* Tidy up method names and calls

* options client

* tweaks

* pr comments

* Update thorclient/common/common.go

Co-authored-by: libotony <[email protected]>

* pr comments

* Adding Subscriptions

* Pr comments

* adjust func orders

* pr comments

* changing subscribe to use the channel close vs multiple channels

* adding go-doc

* no error after unsubscribe

* pr comments

* checking status code is 2xx

* fix: change FilterTransfers argument

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>

* Add testchain package (#844)

* Refactor thor node

* thorchain allows insertion of blocks

* remove thorNode, added testchain

* clean up + comments

* adding license headers

* adding templating tests for thorclient

* Remove test event hacks

* remove types

* removed chain_builder + added logdb to testchain

* pr comments

* Update test/testchain/chain.go

Co-authored-by: libotony <[email protected]>

---------

Co-authored-by: libotony <[email protected]>

* cmd/thor: update instance dir to v4

* trie: implement varint-prefix coder

* deps: add github.com/qianbin/drlp

* trie: implement appendHexToCompact & compactLen

* trie: temporarily remove merkle proof stuff

* trie: many changes

* disk usage reduced by 33% (force embedding shortnode)
* new encoding method for storing nodes
* optimize trie hashing
* versioning standalone nodes
* remove extended trie
* improve trie interface
* simplify NodeIterator, remove unused codes

* trie: optimize full-node encoding/decoding

* trie: tweak shortnode encoding

* muxdb: move engine pkg

* trie: add Version() method for node interface

* muxdb: refactor due to trie updates and:

* remove leafbank stuff
* simplify muxdb.Trie implementation
* improve root node cache using ttl eviction
* add leaf key filter

* chain: a lot of changes

* improve block content storage scheme
* remove steady block tracking
* remove tx & receipt cache

* state: changes due to update of trie

* lowrlp: remove this pkg

* txpool: changes due to underlying pkg update

* genesis: changes due to underlying pkg update

* consensus: changes due to underlying pkg update

* builtin: changes due to underlying pkg update

* runtime: changes due to underlying pkg update

* api: changes due to underlying pkg update

* cmd/thor/pruner: rename pkg optimizer to pruner

* cmd/thor: changes due to underlying pkg update

* muxdb: abandon leaf filter

* cmd/thor/pruner: use smaller period when nearly synced

* muxdb: improve trie node path encoding

* trie: treat short nodes as standalone nodes when skipping hash

* cmd/thor: fix disablePrunerFlag not work

* trie: improve refNode encoding/decoding

* muxdb: improve history node key encoding

* cmd/thor: adjust pruner parameters

* build: fix test cases

* lint: fix lint error

* muxdb: fix ver encoding in node blob cache

* muxdb: add test cases for cache

* runtime: fix test compile error

* make build and test pass after rebase

* add back pruner tests

* add tests for node encoding

* minor typo

* update named store space prefix

* add more tests

* fix block summary in repo

* make build and test pass after rebase

* add back pruner tests

* remove SetBestBlockID from tests

* minor improvement

* pr comments

* adding a comment

* Metrics: Cache hit/miss (#886)

* change

* reverted to previous format

* Add dummy cache for inmem ops (#883)

* Add empty cache for inmem ops

* changing name to dummyCache

* Metrics: Reuse `shouldLog` so we get cache hit/miss data at the same pace (#888)

* first commit

* first commit

* Metrics: Disk IO reads and writes (#890)

* changes

* removed log

* sleeping any way

* pr review

* Pedro/maindb v4/benchmarks (#891)

* Adding Benchmark tests

* processing txs

* Working benchmarks

* lint

* adding tempdir

* improve cache stats log and metric

* totally removed SetBestBlockID

* fix: logs API not returning results when to=0,from=omitted

* minor update

* fix: PR comments + lint

* improve convert range logic

* chore: remove debug log

---------

Co-authored-by: Paolo Galli <[email protected]>
Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>
Co-authored-by: qianbin <[email protected]>
Co-authored-by: Miguel Angel Rojo <[email protected]>
darrenvechain added a commit that referenced this pull request Dec 9, 2024
* fix(documentation): use absolute links in markdown (#889)

* Add benchmark test to node block process (#892)

* Add benchmark test to node block process

* added file-based storage

* use tempdir

* update dependency go-ethereum (#895)

* chore: update API metrics bucket and endpoint names (#893)

* chore: update API metrics bucket and endpoint names

* fix: typo & tests

* fix: lint

* chore: add websocket total counter

* fix: txs endpoints names & ws subject

* fix: unit tests

* chore: standardise naming convention

* chore: add websocke duration & http code

* chore: add websocke duration & http code

* fix: lint issues

* fix: sync issues with metrics

* chore: update websocket durations bucket

* fix: PR comments - use sync.Once

* chore: update builtin generation (#896)

* chore: update builtin generation

* fix: update GHA

* Thor client (#818)

* feat: add thorclient

* refactor: remove roundTripper

* refactor: change null check

* clean: remove commented code

* feat: add account revision and pending tx

* fix: add licence headers and fix linter issue

* refactor: rename package

* refactor: change revision type to string

* refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers

* refactor: change FilterEvents and FilterTransactions request type to EventFilter

* Adding common.EventWrapper to handle channel errors

* tweak

* update rawclient + update account tests

* tidy up names

* update tests

* pr comments

* adding raw tx

* Tidy up method names and calls

* options client

* tweaks

* pr comments

* Update thorclient/common/common.go

Co-authored-by: libotony <[email protected]>

* pr comments

* Adding Subscriptions

* Pr comments

* adjust func orders

* pr comments

* changing subscribe to use the channel close vs multiple channels

* adding go-doc

* no error after unsubscribe

* pr comments

* checking status code is 2xx

* fix: change FilterTransfers argument

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>

* Add testchain package (#844)

* Refactor thor node

* thorchain allows insertion of blocks

* remove thorNode, added testchain

* clean up + comments

* adding license headers

* adding templating tests for thorclient

* Remove test event hacks

* remove types

* removed chain_builder + added logdb to testchain

* pr comments

* Update test/testchain/chain.go

Co-authored-by: libotony <[email protected]>

---------

Co-authored-by: libotony <[email protected]>

* cmd/thor: update instance dir to v4

* trie: implement varint-prefix coder

* deps: add github.com/qianbin/drlp

* trie: implement appendHexToCompact & compactLen

* trie: temporarily remove merkle proof stuff

* trie: many changes

* disk usage reduced by 33% (force embedding shortnode)
* new encoding method for storing nodes
* optimize trie hashing
* versioning standalone nodes
* remove extended trie
* improve trie interface
* simplify NodeIterator, remove unused codes

* trie: optimize full-node encoding/decoding

* trie: tweak shortnode encoding

* muxdb: move engine pkg

* trie: add Version() method for node interface

* muxdb: refactor due to trie updates and:

* remove leafbank stuff
* simplify muxdb.Trie implementation
* improve root node cache using ttl eviction
* add leaf key filter

* chain: a lot of changes

* improve block content storage scheme
* remove steady block tracking
* remove tx & receipt cache

* state: changes due to update of trie

* lowrlp: remove this pkg

* txpool: changes due to underlying pkg update

* genesis: changes due to underlying pkg update

* consensus: changes due to underlying pkg update

* builtin: changes due to underlying pkg update

* runtime: changes due to underlying pkg update

* api: changes due to underlying pkg update

* cmd/thor/pruner: rename pkg optimizer to pruner

* cmd/thor: changes due to underlying pkg update

* muxdb: abandon leaf filter

* cmd/thor/pruner: use smaller period when nearly synced

* muxdb: improve trie node path encoding

* trie: treat short nodes as standalone nodes when skipping hash

* cmd/thor: fix disablePrunerFlag not work

* trie: improve refNode encoding/decoding

* muxdb: improve history node key encoding

* cmd/thor: adjust pruner parameters

* build: fix test cases

* lint: fix lint error

* muxdb: fix ver encoding in node blob cache

* muxdb: add test cases for cache

* runtime: fix test compile error

* make build and test pass after rebase

* add back pruner tests

* add tests for node encoding

* minor typo

* update named store space prefix

* add more tests

* fix block summary in repo

* make build and test pass after rebase

* add back pruner tests

* remove SetBestBlockID from tests

* minor improvement

* pr comments

* adding a comment

* Metrics: Cache hit/miss (#886)

* change

* reverted to previous format

* Add dummy cache for inmem ops (#883)

* Add empty cache for inmem ops

* changing name to dummyCache

* Metrics: Reuse `shouldLog` so we get cache hit/miss data at the same pace (#888)

* first commit

* first commit

* Metrics: Disk IO reads and writes (#890)

* changes

* removed log

* sleeping any way

* pr review

* Pedro/maindb v4/benchmarks (#891)

* Adding Benchmark tests

* processing txs

* Working benchmarks

* lint

* adding tempdir

* improve cache stats log and metric

* totally removed SetBestBlockID

* Maindb v4 Transaction benchmark plus cache (#894)

* Thor client (#818)

* feat: add thorclient

* refactor: remove roundTripper

* refactor: change null check

* clean: remove commented code

* feat: add account revision and pending tx

* fix: add licence headers and fix linter issue

* refactor: rename package

* refactor: change revision type to string

* refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers

* refactor: change FilterEvents and FilterTransactions request type to EventFilter

* Adding common.EventWrapper to handle channel errors

* tweak

* update rawclient + update account tests

* tidy up names

* update tests

* pr comments

* adding raw tx

* Tidy up method names and calls

* options client

* tweaks

* pr comments

* Update thorclient/common/common.go

Co-authored-by: libotony <[email protected]>

* pr comments

* Adding Subscriptions

* Pr comments

* adjust func orders

* pr comments

* changing subscribe to use the channel close vs multiple channels

* adding go-doc

* no error after unsubscribe

* pr comments

* checking status code is 2xx

* fix: change FilterTransfers argument

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>

* Add testchain package (#844)

* Refactor thor node

* thorchain allows insertion of blocks

* remove thorNode, added testchain

* clean up + comments

* adding license headers

* adding templating tests for thorclient

* Remove test event hacks

* remove types

* removed chain_builder + added logdb to testchain

* pr comments

* Update test/testchain/chain.go

Co-authored-by: libotony <[email protected]>

---------

Co-authored-by: libotony <[email protected]>

* cmd/thor: update instance dir to v4

* trie: implement varint-prefix coder

* deps: add github.com/qianbin/drlp

* trie: implement appendHexToCompact & compactLen

* trie: temporarily remove merkle proof stuff

* trie: many changes

* disk usage reduced by 33% (force embedding shortnode)
* new encoding method for storing nodes
* optimize trie hashing
* versioning standalone nodes
* remove extended trie
* improve trie interface
* simplify NodeIterator, remove unused codes

* trie: optimize full-node encoding/decoding

* trie: tweak shortnode encoding

* muxdb: move engine pkg

* trie: add Version() method for node interface

* muxdb: refactor due to trie updates and:

* remove leafbank stuff
* simplify muxdb.Trie implementation
* improve root node cache using ttl eviction
* add leaf key filter

* chain: a lot of changes

* improve block content storage scheme
* remove steady block tracking
* remove tx & receipt cache

* state: changes due to update of trie

* lowrlp: remove this pkg

* txpool: changes due to underlying pkg update

* genesis: changes due to underlying pkg update

* consensus: changes due to underlying pkg update

* builtin: changes due to underlying pkg update

* runtime: changes due to underlying pkg update

* api: changes due to underlying pkg update

* cmd/thor/pruner: rename pkg optimizer to pruner

* cmd/thor: changes due to underlying pkg update

* muxdb: abandon leaf filter

* cmd/thor/pruner: use smaller period when nearly synced

* muxdb: improve trie node path encoding

* trie: treat short nodes as standalone nodes when skipping hash

* cmd/thor: fix disablePrunerFlag not work

* trie: improve refNode encoding/decoding

* muxdb: improve history node key encoding

* cmd/thor: adjust pruner parameters

* build: fix test cases

* lint: fix lint error

* muxdb: fix ver encoding in node blob cache

* muxdb: add test cases for cache

* runtime: fix test compile error

* make build and test pass after rebase

* add back pruner tests

* add tests for node encoding

* minor typo

* update named store space prefix

* add more tests

* fix block summary in repo

* make build and test pass after rebase

* add back pruner tests

* remove SetBestBlockID from tests

* minor improvement

* pr comments

* adding a comment

* Metrics: Cache hit/miss (#886)

* change

* reverted to previous format

* Add dummy cache for inmem ops (#883)

* Add empty cache for inmem ops

* changing name to dummyCache

* Metrics: Reuse `shouldLog` so we get cache hit/miss data at the same pace (#888)

* first commit

* first commit

* Metrics: Disk IO reads and writes (#890)

* changes

* removed log

* sleeping any way

* pr review

* Pedro/maindb v4/benchmarks (#891)

* Adding Benchmark tests

* processing txs

* Working benchmarks

* lint

* adding tempdir

* Adding transactions benchmark + repository cache

* improve cache stats log and metric

* totally removed SetBestBlockID

* removed unused tests

* update bench tests

* getreceipts metrics + lint

---------

Co-authored-by: Paolo Galli <[email protected]>
Co-authored-by: libotony <[email protected]>
Co-authored-by: qianbin <[email protected]>
Co-authored-by: Darren Kelly <[email protected]>
Co-authored-by: Miguel Angel Rojo <[email protected]>

* reduce clauses() allocations

* bug: fix logs endpoints query (#900)

* Thor client (#818)

* feat: add thorclient

* refactor: remove roundTripper

* refactor: change null check

* clean: remove commented code

* feat: add account revision and pending tx

* fix: add licence headers and fix linter issue

* refactor: rename package

* refactor: change revision type to string

* refactor: rename GetLogs and GetTransfers to FilterEvents and FilterTransfers

* refactor: change FilterEvents and FilterTransactions request type to EventFilter

* Adding common.EventWrapper to handle channel errors

* tweak

* update rawclient + update account tests

* tidy up names

* update tests

* pr comments

* adding raw tx

* Tidy up method names and calls

* options client

* tweaks

* pr comments

* Update thorclient/common/common.go

Co-authored-by: libotony <[email protected]>

* pr comments

* Adding Subscriptions

* Pr comments

* adjust func orders

* pr comments

* changing subscribe to use the channel close vs multiple channels

* adding go-doc

* no error after unsubscribe

* pr comments

* checking status code is 2xx

* fix: change FilterTransfers argument

---------

Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>

* Add testchain package (#844)

* Refactor thor node

* thorchain allows insertion of blocks

* remove thorNode, added testchain

* clean up + comments

* adding license headers

* adding templating tests for thorclient

* Remove test event hacks

* remove types

* removed chain_builder + added logdb to testchain

* pr comments

* Update test/testchain/chain.go

Co-authored-by: libotony <[email protected]>

---------

Co-authored-by: libotony <[email protected]>

* cmd/thor: update instance dir to v4

* trie: implement varint-prefix coder

* deps: add github.com/qianbin/drlp

* trie: implement appendHexToCompact & compactLen

* trie: temporarily remove merkle proof stuff

* trie: many changes

* disk usage reduced by 33% (force embedding shortnode)
* new encoding method for storing nodes
* optimize trie hashing
* versioning standalone nodes
* remove extended trie
* improve trie interface
* simplify NodeIterator, remove unused codes

* trie: optimize full-node encoding/decoding

* trie: tweak shortnode encoding

* muxdb: move engine pkg

* trie: add Version() method for node interface

* muxdb: refactor due to trie updates and:

* remove leafbank stuff
* simplify muxdb.Trie implementation
* improve root node cache using ttl eviction
* add leaf key filter

* chain: a lot of changes

* improve block content storage scheme
* remove steady block tracking
* remove tx & receipt cache

* state: changes due to update of trie

* lowrlp: remove this pkg

* txpool: changes due to underlying pkg update

* genesis: changes due to underlying pkg update

* consensus: changes due to underlying pkg update

* builtin: changes due to underlying pkg update

* runtime: changes due to underlying pkg update

* api: changes due to underlying pkg update

* cmd/thor/pruner: rename pkg optimizer to pruner

* cmd/thor: changes due to underlying pkg update

* muxdb: abandon leaf filter

* cmd/thor/pruner: use smaller period when nearly synced

* muxdb: improve trie node path encoding

* trie: treat short nodes as standalone nodes when skipping hash

* cmd/thor: fix disablePrunerFlag not work

* trie: improve refNode encoding/decoding

* muxdb: improve history node key encoding

* cmd/thor: adjust pruner parameters

* build: fix test cases

* lint: fix lint error

* muxdb: fix ver encoding in node blob cache

* muxdb: add test cases for cache

* runtime: fix test compile error

* make build and test pass after rebase

* add back pruner tests

* add tests for node encoding

* minor typo

* update named store space prefix

* add more tests

* fix block summary in repo

* make build and test pass after rebase

* add back pruner tests

* remove SetBestBlockID from tests

* minor improvement

* pr comments

* adding a comment

* Metrics: Cache hit/miss (#886)

* change

* reverted to previous format

* Add dummy cache for inmem ops (#883)

* Add empty cache for inmem ops

* changing name to dummyCache

* Metrics: Reuse `shouldLog` so we get cache hit/miss data at the same pace (#888)

* first commit

* first commit

* Metrics: Disk IO reads and writes (#890)

* changes

* removed log

* sleeping any way

* pr review

* Pedro/maindb v4/benchmarks (#891)

* Adding Benchmark tests

* processing txs

* Working benchmarks

* lint

* adding tempdir

* improve cache stats log and metric

* totally removed SetBestBlockID

* fix: logs API not returning results when to=0,from=omitted

* minor update

* fix: PR comments + lint

* improve convert range logic

* chore: remove debug log

---------

Co-authored-by: Paolo Galli <[email protected]>
Co-authored-by: otherview <[email protected]>
Co-authored-by: libotony <[email protected]>
Co-authored-by: qianbin <[email protected]>
Co-authored-by: Miguel Angel Rojo <[email protected]>

* chore(chain): add repo cache metrics (#910)

* chore(chain): add repo cache metrics

* refactor(chain): cache hit miss

---------

Co-authored-by: Darren Kelly <[email protected]>
Co-authored-by: Pedro Gomes <[email protected]>
Co-authored-by: Paolo Galli <[email protected]>
Co-authored-by: qianbin <[email protected]>
Co-authored-by: Miguel Angel Rojo <[email protected]>
Co-authored-by: Darren Kelly <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants