Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Direct proptest support #1220

Open
brson opened this issue Feb 2, 2024 · 0 comments
Open

Direct proptest support #1220

brson opened this issue Feb 2, 2024 · 0 comments

Comments

@brson
Copy link
Contributor

brson commented Feb 2, 2024

What problem does your feature solve?

Right now we have direct support for fuzz testing via cargo-fuzz and proptesting via proptest-arbitrary-interop. This support mainly consists of implementations of Arbitrary, the trait required by cargo-fuzz, for Soroban types. proptest-arbitrary-interop can be used to convert between these implementations and the traits required by proptest.

There are two main problems with this arrangement:

  1. The cargo-fuzz UX is poor - fuzz tests are their own programs, the code doesn't look like or live next to unit tests, and the test runner output is unsophisticated. I believe the test runner output is built into libfuzzer, which receives no updates, and not under the control of cargo-fuzz.
  2. The proptests enabled by this scheme are limited - proptest allows for many "strategies" for creating and minimizing test inputs, but this scheme only allows for random number inputs that can't sensibly be minimized. cc Improve fidelity of Arbitrary-based proptests #1105

What would you like to see?

It may be worth implementing direct support for the proptest crate, which would entail implementing different traits for Soroban types, and again might require a complex prototype/builder pattern to construct those types into Soroban Envs.

This would increase the maintenance requirements for the SDK, and make the fuzz/prop testing story more complex. I do find proptest quite complicated, needing to read significant amounts of the proptest book to understand how to use it.

What alternatives are there?

Instead of proptests, https://github.com/loiclec/fuzzcheck-rs is a fuzzing engine that allows fuzz tests to be written more like Rust unit tests, inline with other code. Similar experience with a different testing strategy. Less mature than proptest. This also requires supporting different traits than the existing Arbitrary.

The quickcheck crate is similar and may be less complex to understand than the proptest crate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants
@brson and others