feat(forge): integration testing & test harnesses

[OliverNChalk]

Component

Forge

Describe the feature you would like

Current Behavior

• forge test picks up and runs all function test*
• This means that both unit tests and integration tests are run when forge test is called
• You cannot support integration tests across two networks without filtering using various match calls
• Generally, this leads to projects using a Makefile to discriminate between tests

Desired Behavior

Ideally, we would distinguish between two types of tests:

• unit test: runs with no environmental dependency and is deterministic
• integration test: expects a specific environment and may not be deterministic

Unit testing would continue as it currently runs, the UX of forge is strong in this area. Integration tests would however carry around config as to the environment they expect to run in. I see two ways to achieve this:

• Declare config in solidity
  • Support setting RPC URL / chain-id (where it then reads rpc-url from environment/config) as a minimum
  • Inherit from TestHarness that requires config to be passed in constructor
  • Set desired config using cheatcodes
• Declare config in toml/json
  • Support setting RPC URL / chain-id (where it then reads rpc-url from environment/config) as a minimum
  • Ideally support passing arguments to setUp / discrete functions within the test contract
  • Pin config to a contract or function within the contract
  • Could have multiple configs per contract or function, enabling a form of table testing (though I suspect there might be better UX from pure solidity here)
  • Would enable running new contracts through re-usable tests, as you could potentially pass the contract/bytecode as an argument for testing. E.g. an ERC-20 test harness that gets run against multiple contracts/deployments

Next steps

Interested in getting feedback on:

• The demand for this sort of functionality
• Whether the potential solutions are appealing
• Alternative approaches

Additional context

No response

[onbjerg]

Can you expand on ow this differs from fork tests?

Test discrimination by profile is coming in #1378, which also means you can set different fork urls and blocks for different sets of tests. For cross-network tests in the same test, see #834. For table testing see #858

[OliverNChalk]

Can you expand on ow this differs from fork tests?

Test discrimination by profile is coming in #1378, which also means you can set different fork urls and blocks for different sets of tests. For cross-network tests in the same test, see #834. For table testing see #858

I think the harness testing would be achieved via #858 and #834, each case would declare its parameters/environment including fork_url.

As for #1378, I opened this issue (#1414) as a Quality of Life change & to make it more intuitive for new users.

[onbjerg]

Then in that case we can close this?

[OliverNChalk]

Yep, the only difference is declaring test cases in solidity vs some other format. Happy to close this for now and watch how the other issues play out