Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: flaky gas snapshot for some fuzz tests #7942

Closed
2 tasks done
cruzdanilo opened this issue May 17, 2024 · 2 comments · Fixed by #7951
Closed
2 tasks done

bug: flaky gas snapshot for some fuzz tests #7942

cruzdanilo opened this issue May 17, 2024 · 2 comments · Fixed by #7951
Labels
T-bug Type: bug

Comments

@cruzdanilo
Copy link
Contributor

Component

Forge

Have you ensured that all of these are up to date?

  • Foundry
  • Foundryup

What version of Foundry are you on?

forge 0.2.0 (467aff3 2024-05-17T00:17:24.004804000Z)

What command(s) is the bug in?

forge snapshot

Operating System

macOS (Apple Silicon)

Describe the bug

for some fuzz tests (not sure which ones), forge fails to generate consistent gas snapshots, even when using the --force flag. this prevents it from being relied upon for ci validation (forge snapshot --check). the same behavior happens on macos (apple silicon) and linux (x86-64).

i created a somewhat minimal reproduction repository here: https://github.com/cruzdanilo/foundry-snapshot-repro

this is one output generated by for i in {1..16}; do forge snapshot --force | grep runs; done

[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407539, ~: 408263)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407539, ~: 408263)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407539, ~: 408263)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407539, ~: 408263)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407539, ~: 408263)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407539, ~: 408263)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407518, ~: 408251)

this was run on github ci, here is the report.

@cruzdanilo cruzdanilo added the T-bug Type: bug label May 17, 2024
@grandizzy
Copy link
Collaborator

grandizzy commented May 20, 2024

I run the test and can see 2 different values for fuzzed UserOperation input, e.g. in 2 different runs showing

[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 417326, ~: 417135)
[PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 417308, ~: 417135)

there are 2 different fuzzed cased values, run 164

UserOperation {
    0x00000000000000000000000000000000000000000000000000000000000022e9,
    0x0000000000000000000000000000000000000000000000000000000000002733,
    9764,
    12673873082346130924691454452779514193164883897088292420374917853190248779330,
    10006
}

vs

UserOperation {
    0x00000000000000000000000000000000000000000000000000000000000022e9,
    0x0000000000000000000000000000000000000000000000000000000000002733,
    9764,
    18542991761951108740563055453066386026290576689311603472268584080832751656013,
    10006
}

and run 244

UserOperation {
    0x0000000000000000000000000000000000000000000000000000000000002733,
    0x0000000000000000000000000000000000000000000000000000000000000e69,
    1638867016,
    801,
    12673873082346130924691454452779514193164883897088292420374917853190248779331
}

vs

UserOperation {
    0x0000000000000000000000000000000000000000000000000000000000002733,
    0x0000000000000000000000000000000000000000000000000000000000000e69,
    1638867016,
    801,
    18542991761951108740563055453066386026290576689311603472268584080832751656014
}

which explains why the difference

Similar issue is reported in
#5199
#5689
#6179

@grandizzy
Copy link
Collaborator

@cruzdanilo this happens due to fuzzing from mutated state values which are not deterministic, if adding locally

    /// forge-config: default.fuzz.dictionary-weight = 0
    function test_snapshot(UserOperation calldata userOp) external {

results are always [PASS] test_snapshot((bytes,bytes,uint256,uint256,uint256)) (runs: 256, μ: 407510, ~: 407539)

Can you pls try and check if consistent results?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T-bug Type: bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants