Tests run twice if you change compiler version

[leonardoalt]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.2.0 (e85435a 2022-04-14T00:06:25.800261822+00:00)

What command(s) is the bug in?

forge test

Operating System

Linux

Describe the bug

Steps:

1. Run forge init

2. Change the Solidity pragma version to >=0.8.10 in src/Contract.sol and src/test/Contract.t.sol

3. Add solc = "0.8.10" to foundry.toml

4. Run forge test -vvvv, you'll see

Running 1 test for src/test/Contract.t.sol:ContractTest
[PASS] testExample() (gas: 190)
Traces:
  [190] ContractTest::testExample() 
    └─ ← ()

5. Replace the solc version in foundry.toml by solc = "0.8.13"

6. Run forge test -vvvv, you'll now see

Running 1 test for src/test/Contract.t.sol:ContractTest
[PASS] testExample() (gas: 190)
Traces:
  [190] ContractTest::testExample() 
    └─ ← ()

Test result: ok. 1 passed; 0 failed; finished in 566.01µs

Running 1 test for src/test/Contract.t.sol:ContractTest
[PASS] testExample() (gas: 190)
Traces:
  [190] ContractTest::testExample() 
    └─ ← ()

So it looks like the tests are running twice, I think once with each compiler version from the two mentioned above. Not sure if this behavior is intended, but I'd guess not.

[onbjerg]

@mattsse Maybe this is because we have a cached version of the test file, and a new non-cached version of the test file and we see both and run both?

[mattsse]

that could be it, working a repro