Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use the presence of property-based testing in Haskell to detect fuzzing #2830

Closed
chungyc opened this issue Apr 6, 2023 · 3 comments · Fixed by #2843
Closed

Use the presence of property-based testing in Haskell to detect fuzzing #2830

chungyc opened this issue Apr 6, 2023 · 3 comments · Fixed by #2843
Assignees
Labels
check/Fuzzing kind/enhancement New feature or request

Comments

@chungyc
Copy link
Contributor

chungyc commented Apr 6, 2023

Is your feature request related to a problem? Please describe.

Property-based testing is widespread with Haskell projects. They either generate test instances randomly or exhaustively and test that specific properties are satisfied. This is pretty much fuzzing and should be treated as such.

Describe the solution you'd like

QuickCheck, SmallCheck, Hedgehog, and validity are property-based testing frameworks for Haskell that I know of (likely not an exhaustive list).

Detecting their use could be done by scanning for imports of Haskell modules from these Haskell packages in test code, which would be roughly on par with what fuzzing detection for Go does. This can be an entry in languageFuzzSpecs as with Go.

Describe alternatives you've considered

An alternative would be to detect the use of property-based testing Haskell packages in Cabal or hpack files. However, this may have a much higher false positive or false negative rate given the limited matching based on regular expressions that is available. Import statements in Haskell have a much more regular form that would be more easily accommodated with regular expressions.

Pattern matching on specific function declarations will not work because there is no standardized function declaration convention for property-based tests; individual tests may not even be in their own named function.

Additional context

This is basically declaring my intention and willingness to add this detection to fuzzing.go unless the owners think otherwise.

@chungyc chungyc added the kind/enhancement New feature or request label Apr 6, 2023
@naveensrinivasan
Copy link
Member

@chungyc Thanks! Would you like to open a PR to address this?

@chungyc
Copy link
Contributor Author

chungyc commented Apr 10, 2023

I have opened #2843 as a draft. It is not ready for review or merging yet, but it is enough to show the direction I am pursuing.

@chungyc
Copy link
Contributor Author

chungyc commented Apr 10, 2023

#2843 is ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
check/Fuzzing kind/enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants