Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cheatcodes): vm.skip(bool) for skipping tests #5205

Merged
merged 16 commits into from
Jun 28, 2023
Merged

Conversation

Evalir
Copy link
Member

@Evalir Evalir commented Jun 23, 2023

Motivation

Closes #1123

Solution

Adds a vm.skip(bool) cheatcode that marks tests as skipped. It also moves the test runner to use a TestStatus enum to mark success, failure, or a skipped test, which is the bulk of the work here really.

  • On normal tests, this will just check if it failed due to a skip.
  • On fuzz tests, it will fuzz with only one run and check the revert reason.
  • On invariant tests, it will run once and check if it failed due to a skip, like a normal test.

One thing I noticed is that the handling and return types between normal/invariant/fuzz tests are pretty different. When we revisit the test runner we should try and streamline the handling of all of these so we can properly detect EVM errors in an easier way.

TODO:

  • Implement vm.skip(bool)
  • Works on normal tests
  • Works on fuzz tests
  • Works on invariant tests
  • Make magic bytes detection and marking as skipped cleaner by not repeating code

Sample output:
Screenshot 2023-06-27 at 23 54 07

@Evalir Evalir requested review from mattsse and mds1 June 28, 2023 03:50
@Evalir Evalir marked this pull request as ready for review June 28, 2023 03:50
@Evalir Evalir changed the title [WIP] feat(cheatcodes): vm.skip(bool) for skipping tests feat(cheatcodes): vm.skip(bool) for skipping tests Jun 28, 2023
Copy link
Member

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Evalir
Copy link
Member Author

Evalir commented Jun 28, 2023

@mds1 I think we'll merge this initial version that only works at the test level to keep test runner changes minimal—adding a reason string or making it work correctly at the set up level could involve moving a bit more code and I'd like to add this incrementally in another PR

@Evalir Evalir merged commit ac5d367 into master Jun 28, 2023
@Evalir Evalir deleted the evalir/vm-skip branch June 28, 2023 15:23
@shazow
Copy link

shazow commented Jun 28, 2023

Thank you for working on this. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Function Modifier or an Opcode to mark a test as skipped conditionally
3 participants