meta(fuzzer): tracking issue for improvements to property tests and invariant tests

[mds1]

Component

Forge

Describe the feature you would like

This issue tracks general fuzzer improvements that improve both property tests and invariant tests. (There will be a separate tracking issue for things that are strictly invariant test related).

Items are roughly ordered by my opinion of priority, both at the header-level and the bullet-level. I say "roughly ordered" because some items are much bigger scope than others so having a strict ordering doesn't make much sense.

Fuzzer Benchmark/Tests

First priority is some tests and benchmarks. to verify behavior of the fuzzer has not regressed as it's changed. Potential test cases:

• condition not caught by invariant/fuzzer #2851
• https://github.com/crytic/echidna/tree/master/tests/solidity
• https://blog.trailofbits.com/2023/02/27/reusable-properties-ethereum-contracts-echidna/

This is tracked in a separate issue here: #3411

Refactor / Tech Debt / Cleanup

• Remove proptest? It has some limitations and isn't currently maintained (ref discussions in feat: support time-based and continuous fuzzing and invariant testing  #990 and Is Proptest abandoned? proptest-rs/proptest#268). This would be a big change so shouldn't be done until we have tests/benchmarks in place.
• Deprecation of fuzz.max_global_rejects config #3153

Features / Performance

High Priority

• feat (proposal): UX for granularity over fuzz, invariant, and solidity sections #3062 OR Selective fuzz runs with NatSpec comments "@custom:fuzz-runs NUMBER" #4085 — Before implementing, need to decide if we like the config file approach or a comment-based approach, then finalize UX, then implement. Comment approach does add flexibility and allows future expansion to things like Ability to get sorted arrays when fuzzing #4097
• feat: deduplicate fuzz inputs #3521
• feat: add real-time counter for fuzz/invariant runs #585

Low priority:

• feat: add config option to turn off shrinking #4867
• feat: support time-based and continuous fuzzing and invariant testing  #990 — Not supported by proptest so currently hard, but may be easy after a refactor. Low priority because it can be replicated by setting runs to max uint
• vm.canRevert(): allow tests to revert in expected cases #4090
• Ability to get sorted arrays when fuzzing #4097 — If we go with the comment approach for #3062, this is doable
• Console logs should ideally print _during_ fuzzing, not just after testing is complete #3844 — I think this would be hard to implement with the current architecture since things run in parallel, and you can accomplish the same thing with vm.writeLine, so this isn't necessary IMO. May be worth closing
• max_test_reject_rate: set a maximum test rejection rate per test function #4091 — This flag would conflict with the existing max_test_rejects flag, so one needs to take precedence and we need warn if the both exist. Personally this feels like the complexity/config growth isn't worth it since the same functionality can already be accomplished, so may be worth closing

Failure Persistence and Replay

I have this last because the functionality can be replicated a bit tediously by copying failed tests into concrete tests.

• feat: fuzz failure persistence #2551
• feat: fuzz corpus saving and replay in standard format #2552

Long Term Fuzz Techniques

Some more advanced techniques to consider down the road:

• Using outputs from symbolic execution to seed the fuzzer
• Techniques described in the discussion in Progress on Invariant Tests #3277

Additional context

No response

[zerosnacks]

Closing in favor of the active #8076 fuzzer meta ticket

Made sure that all unresolved tickets have been incorporated (this was already the case) with the exception of #4090, this was moved to the cheatcodes meta: #4439