Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(forge): selectively enable Etherscan trace resolving when a test in ran in a forked environment and return block number in trace on a failed test #7606

Closed
wants to merge 52 commits into from

Conversation

zerosnacks
Copy link
Member

@zerosnacks zerosnacks commented Apr 9, 2024

Motivation

Closes #7555 (bug)
Closes #7312 (bug)
Closes #7574 (feature)

In short, when an ETHERSCAN_API_KEY is defined in a users' .env it would trigger Etherscan trace resolving for local tests which would result in a massive slowdown after running into rate limiting.

Solution

Implements a way to mark test results as whether having ran in a forked or non-forked environment. This is then used to selectively enable or disable Etherscan tracing (without re-initializing).

As requested in #7574, upon a encountering a test failure in a forked test it will now display the fork block number. On succesful runs it is not displayed.

Known limitations:

  • Generally unable to differentiate remote forks and local forks
  • When using vm.roll(n) local forks are created

Given that trace caching is now resolved these limitations are not that big of a deal, at worst the experience is as bad as it is now but only for the initial run.

The chosen approach was to add high-level context tracking which can easily be expanded to track additional call context related information.

@onbjerg onbjerg added the T-bug Type: bug label Apr 9, 2024
Copy link
Member

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good direction

crates/forge/bin/cmd/test/mod.rs Outdated Show resolved Hide resolved
Comment on lines 488 to 490
/// Various types of test environments
#[derive(Clone, Debug, Serialize, Deserialize)]
pub enum TestEnvironment {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this, this makes it easier to add more modes

mattsse pushed a commit to foundry-rs/block-explorers that referenced this pull request Apr 15, 2024
Closes: foundry-rs/foundry#6792

Due to a small logical error and a missing smoke test the Etherscan
cache was constantly being invalidated.

On a large codebase with 197 tests, without this optimization:
foundry-rs/foundry#7606

Comparison:

1. Ran 17 test suites in 53.41s (64.87s CPU time): 197 tests passed, 0
failed, 0 skipped (197 total tests) (no cache)
2. Ran 17 test suites in 6.29s (49.10s CPU time): 197 tests passed, 0
failed, 0 skipped (197 total tests) (with cache)
Copy link
Member

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some questions about ForkContext vs TestEnvironement

Comment on lines +10 to +13
pub struct ContextCollector {
/// The collected execution contexts.
pub contexts: Vec<Context>,
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need an inspector for this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was added as an inspector so that it would run on every call occurring inside the function body, that way we were able to track the modifications made to the environment (such as the updating of the block number).

It isn't strictly necessary as we only need the most recent change to the block env, excluding a reset to genesis

crates/forge/src/runner.rs Outdated Show resolved Hide resolved
Comment on lines +17 to +32
let block_number = ecx.inner.env.block.number;

// Skip if the block number is at genesis
if block_number == U256::from(1) {
return None;
}

// Skip if the previous context is the same
if let Some(Context { block_number: prev_block_number }) = self.contexts.last() {
if *prev_block_number == block_number {
return None;
}
}

// Push the new context
self.contexts.push(Context { block_number });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, not sure this will work, because there are also cheatcodes that modify the block number.

the fork context is essentially already tracked in the Backend internally.
can we instead extract this after executing a test in the testresult?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cheatcodes overwrite the env of the backend referenced here

/// Note: in case there are any cheatcodes executed that modify the environment, this will
/// update the given `env` with the new values.

By extracting it after the test we can only capture the last context as the backend overwrites the env, sufficient for the current usecase

@zerosnacks zerosnacks changed the title fix(forge): selectively enable Etherscan trace resolving when a test in ran in a forked environment fix(forge): selectively enable Etherscan trace resolving when a test in ran in a forked environment and return block number on a failed test Apr 30, 2024
@zerosnacks zerosnacks changed the title fix(forge): selectively enable Etherscan trace resolving when a test in ran in a forked environment and return block number on a failed test fix(forge): selectively enable Etherscan trace resolving when a test in ran in a forked environment and return block number in trace on a failed test Apr 30, 2024
@zerosnacks
Copy link
Member Author

Closing as stale as the PR as it is up now is insufficient

#7574 is still relevant and can be picked up separately

Conditional Etherscan trace resolving not worth spending more time on given the cache busting fix in foundry-rs/block-explorers#40

@zerosnacks zerosnacks closed this Jun 25, 2024
@zerosnacks zerosnacks deleted the zerosnacks/fix-tracing-regression branch June 25, 2024 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T-bug Type: bug
Projects
None yet
3 participants