Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the performance of eth_getLogs #568

Closed
wants to merge 5 commits into from

Conversation

JukLee0ira
Copy link

@JukLee0ira JukLee0ira commented Jul 1, 2024

Proposed changes

Not all fields of a receipt are persisted to disk. Some are re-computed on read. The most expensive field to re-compute is the bloom filter which will not even be returned in eth_getLogs. Here we define a new decoding of receipts specifically for this use-case which avoids extra processing.

Test 'eth_getLogs'

Vertical comparison

Before optimization, run the command and save the result to file1.log:

curl -X POST -H "Content-Type: application/json" --data '{  "method":"eth_getLogs",  "params":[{ "fromBlock":"0x1010d", "toBlock":"0x11a400",    address":"xdc0000000000000000000000000000000000000088",    "topics":[ ]  }],  "id":1,  "jsonrpc":"2.0"}' http://127.0.0.1:8545 | jq > file1.log

After optimization, run the same command and save the result to file2.log:

curl -X POST -H "Content-Type: application/json" --data '{  "method":"eth_getLogs",  "params":[{ "fromBlock":"0x1010d", "toBlock":"0x11a400",    address":"xdc0000000000000000000000000000000000000088",    "topics":[ ]  }],  "id":2,  "jsonrpc":"2.0"}' http://127.0.0.1:8545| jq > file2.log

Here are their diff results:

3c3
<   "id": 1,
---
>   "id": 2,

Horizontal comparison

After the modification, sync the block data, run the command, and save the result to file3.log:

curl -X POST -H "Content-Type: application/json" --data '{  "method":"eth_getLogs",  "params":[{ "fromBlock":"0x1010d", "toBlock":"0x11a400",    address":"xdc0000000000000000000000000000000000000088",    "topics":[ ]  }],  "id":3,  "jsonrpc":"2.0"}' http://127.0.0.1:8545 | jq > file3.log

Run the command to read the log from the public RPC node and save the result to file4.log:

curl -X POST -H "Content-Type: application/json" --data '{  "method":"eth_getLogs",  "params":[{ "fromBlock":"0x1010d", "toBlock":"0x11a400",    address":"xdc0000000000000000000000000000000000000088",    "topics":[ ]  }],  "id":4,  "jsonrpc":"2.0"}' https://apothem.xdcrpc.com | jq > file4.log

Here are their diff results:

3c3
<   "id": 3,
---
>   "id": 4,

Types of changes

What types of changes does your code introduce to XDC network?
Put an in the boxes that apply

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update (if none of the other choices apply)
  • Regular KTLO or any of the maintaince work. e.g code style
  • CICD Improvement
  • Performance

Impacted Components

Which part of the codebase this PR will touch base on,

Put an in the boxes that apply

  • Consensus
  • Account
  • Network
  • Geth
  • Smart Contract
  • External components
  • Not sure (Please specify below)

Checklist

Put an in the boxes once you have confirmed below actions (or provide reasons on not doing so) that

  • This PR has sufficient test coverage (unit/integration test) OR I have provided reason in the PR description for not having test coverage
  • Provide an end-to-end test plan in the PR description on how to manually test it on the devnet/testnet.
  • Tested the backwards compatibility.
  • Tested with XDC nodes running this version co-exist with those running the previous version.
  • Relevant documentation has been updated as part of this PR
  • N/A

@gzliudan
Copy link
Collaborator

gzliudan commented Jul 2, 2024

Please rebase on the latest dev-upgrade branch.

@JukLee0ira JukLee0ira force-pushed the opt-getLogs branch 2 times, most recently from 55fa685 to e680e9d Compare July 2, 2024 02:31
@JukLee0ira JukLee0ira marked this pull request as draft July 2, 2024 02:50
@JukLee0ira JukLee0ira marked this pull request as ready for review July 2, 2024 02:50
@gzliudan gzliudan self-requested a review July 2, 2024 03:33
@JukLee0ira JukLee0ira force-pushed the opt-getLogs branch 11 times, most recently from 300e89d to add2081 Compare July 4, 2024 02:18
Copy link
Collaborator

@gzliudan gzliudan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gzliudan gzliudan self-requested a review July 9, 2024 07:39
@JukLee0ira JukLee0ira force-pushed the opt-getLogs branch 4 times, most recently from 9a90ac8 to f590460 Compare July 9, 2024 11:17
@JukLee0ira JukLee0ira changed the title avoid unnecessary receipt processing for log filtering use lightweight accessor for log filtering and fix deadlock issues Jul 9, 2024
@JukLee0ira JukLee0ira changed the title use lightweight accessor for log filtering and fix deadlock issues Improve the performance of log filtering Jul 9, 2024
@gzliudan gzliudan changed the title Improve the performance of log filtering Improve the performance of eth_getLogs Jul 9, 2024
@JukLee0ira JukLee0ira force-pushed the opt-getLogs branch 3 times, most recently from a1c7160 to 398af71 Compare July 11, 2024 05:28

return api
}

// timeoutLoop runs every 5 minutes and deletes filters that have not been recently used.
// Tt is started when the api is created.
func (api *PublicFilterAPI) timeoutLoop() {
func (api *FilterAPI) timeoutLoop(timeout time.Duration) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new parameter timeout is not used. Please refer to: ethereum#22178

@JukLee0ira JukLee0ira closed this Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants