Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve eth_getLogs performance for latest block #5680

Closed
mattsse opened this issue Dec 4, 2023 · 12 comments
Closed

Improve eth_getLogs performance for latest block #5680

mattsse opened this issue Dec 4, 2023 · 12 comments
Assignees
Labels
A-rpc Related to the RPC implementation C-enhancement New feature or request D-good-first-issue Nice and easy! A great choice to get started

Comments

@mattsse
Copy link
Collaborator

mattsse commented Dec 4, 2023

Describe the feature

Currently we're computing the range of matching blocks of a log filter:

let from = from_block
.map(|num| self.inner.provider.convert_block_number(num))
.transpose()?
.flatten();
let to = to_block
.map(|num| self.inner.provider.convert_block_number(num))
.transpose()?
.flatten();
logs_utils::get_filter_block_range(from, to, start_block, info)

and then request that range:

// loop over the range of new blocks and check logs if the filter matches the log's bloom
// filter
for (from, to) in
BlockRangeInclusiveIter::new(from_block..=to_block, self.max_headers_range)
{
let headers = self.provider.headers_range(from..=to)?;

this is very inefficient for filters such as [latest,latest] which are only interested in the latest blocks, because this range query always hits the disk when the data is always cached (EthStateCache).

so we should improve this
check the range arguments

async fn get_logs_in_block_range(
&self,
filter: &Filter,
from_block: u64,
to_block: u64,
) -> Result<Vec<Log>, FilterError> {

if this is a single block and Filter's upperbound is Latest use the state cache to fetch the block's data

like we do for the block hash variant:

FilterBlockOption::AtBlockHash(block_hash) => {
let mut all_logs = Vec::new();
// all matching logs in the block, if it exists
if let Some((block, receipts)) =
self.eth_cache.get_block_and_receipts(block_hash).await?
{

The get_logs_in_block_range function argument likely needs additional info now (for the currently latest block):

let info = self.inner.provider.chain_info()?;
let best_number = info.best_number;

cc @Arindam2407

Additional context

No response

@mattsse mattsse added C-enhancement New feature or request S-needs-triage This issue needs to be labelled labels Dec 4, 2023
@mattsse mattsse added D-good-first-issue Nice and easy! A great choice to get started A-rpc Related to the RPC implementation and removed S-needs-triage This issue needs to be labelled labels Dec 4, 2023
@mattsse
Copy link
Collaborator Author

mattsse commented Dec 4, 2023

@Arindam2407 what we should do here is implement multiple functions that fetch the range:

  1. via the existing loop over BlockRangeInclusiveIter
  2. single call to cache if range is one block
  3. multiple calls to cache if from..=to is smaller than x blocks (perhaps 10) and their distance from latest is lower than the ethcache's targeted cache range

lmk if anything's unclear

@leontiad
Copy link
Contributor

leontiad commented Dec 8, 2023

@mattsse So the cache calls should be there only in those cases:

  1. from==to==latest
  2. from-to<= threshold and |latest-(from-to)|<=ethcache-range,
    I guess the ethcache-range is max_blocks of EthStateCacheConfig ?

@mattsse
Copy link
Collaborator Author

mattsse commented Dec 8, 2023

yes, because we assume here that the targeted block is still cached if it's near the tip.

I think we can use max_blocks or max_blocks / 2 as a cutoff

@mattsse
Copy link
Collaborator Author

mattsse commented Dec 8, 2023

@Arindam2407 turns out @leontiad started working on this already
so feel free to continue with #5677, sorry about the confusion

@yash-atreya
Copy link
Contributor

@mattsse I can take this and #5677 if @leontiad and @Arindam2407 decide to discontinue working.

@leontiad
Copy link
Contributor

I am on that - opening a WIP PR

@allnil
Copy link
Contributor

allnil commented Jan 30, 2024

take a look

@leontiad
Copy link
Contributor

leontiad commented Jan 31, 2024

Can't push my branch, am I authed to do that? @mattsse

h3rc@~/projects/reth $     git push --set-upstream origin eth_getLogs_cache_improvements
Username for 'https://github.com': leontiad
Password for 'https://[email protected]': 
remote: Permission to paradigmxyz/reth.git denied to leontiad.
fatal: unable to access 'https://github.com/paradigmxyz/reth.git/': The requested URL returned error: 403

@allnil
Copy link
Contributor

allnil commented Jan 31, 2024

@leontiad
Hi Iraklis! I think you cloned reth repo but...you need make a fork, make branch in your forked reth and than you will be able to open merge request which appeared in this repo :)
hope it helps!

@leontiad
Copy link
Contributor

@leontiad Hi Iraklis! I think you cloned reth repo but...you need make a fork, make branch in your forked reth and than you will be able to open merge request which appeared in this repo :) hope it helps!

Thanks!That make it work!

@sambacha
Copy link

sambacha commented Feb 13, 2024

Just wanting some clarification, this latest is really an alias for unsafe correct? 

unsafe being "The most recent block in the canonical chain observed by the client this block can be re-orged out of the canonical chain"

Also @mattsse would you be open to adding blockTimestamp as part of the logs object returned by eth_getLogs1

Footnotes

  1. See this proposal on Ethereum magicians forums regarding adding blockTimestamp to eth_getLogs

@mattsse
Copy link
Collaborator Author

mattsse commented Feb 23, 2024

unsafe being "The most recent block in the canonical chain observed by the client this block can be re-orged out of the canonical chain"

yeah latest is the highest canonical block processed by the chain

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-rpc Related to the RPC implementation C-enhancement New feature or request D-good-first-issue Nice and easy! A great choice to get started
Projects
Archived in project
Development

No branches or pull requests

5 participants