You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now we have direct support for fuzz testing via cargo-fuzz and proptesting via proptest-arbitrary-interop. This support mainly consists of implementations of Arbitrary, the trait required by cargo-fuzz, for Soroban types. proptest-arbitrary-interop can be used to convert between these implementations and the traits required by proptest.
There are two main problems with this arrangement:
The cargo-fuzz UX is poor - fuzz tests are their own programs, the code doesn't look like or live next to unit tests, and the test runner output is unsophisticated. I believe the test runner output is built into libfuzzer, which receives no updates, and not under the control of cargo-fuzz.
The proptests enabled by this scheme are limited - proptest allows for many "strategies" for creating and minimizing test inputs, but this scheme only allows for random number inputs that can't sensibly be minimized. cc Improve fidelity of Arbitrary-based proptests #1105
What would you like to see?
It may be worth implementing direct support for the proptest crate, which would entail implementing different traits for Soroban types, and again might require a complex prototype/builder pattern to construct those types into Soroban Envs.
This would increase the maintenance requirements for the SDK, and make the fuzz/prop testing story more complex. I do find proptest quite complicated, needing to read significant amounts of the proptest book to understand how to use it.
What alternatives are there?
Instead of proptests, https://github.com/loiclec/fuzzcheck-rs is a fuzzing engine that allows fuzz tests to be written more like Rust unit tests, inline with other code. Similar experience with a different testing strategy. Less mature than proptest. This also requires supporting different traits than the existing Arbitrary.
The quickcheck crate is similar and may be less complex to understand than the proptest crate.
The text was updated successfully, but these errors were encountered:
What problem does your feature solve?
Right now we have direct support for fuzz testing via cargo-fuzz and proptesting via proptest-arbitrary-interop. This support mainly consists of implementations of
Arbitrary
, the trait required by cargo-fuzz, for Soroban types. proptest-arbitrary-interop can be used to convert between these implementations and the traits required by proptest.There are two main problems with this arrangement:
What would you like to see?
It may be worth implementing direct support for the
proptest
crate, which would entail implementing different traits for Soroban types, and again might require a complex prototype/builder pattern to construct those types into SorobanEnv
s.This would increase the maintenance requirements for the SDK, and make the fuzz/prop testing story more complex. I do find
proptest
quite complicated, needing to read significant amounts of the proptest book to understand how to use it.What alternatives are there?
Instead of proptests, https://github.com/loiclec/fuzzcheck-rs is a fuzzing engine that allows fuzz tests to be written more like Rust unit tests, inline with other code. Similar experience with a different testing strategy. Less mature than proptest. This also requires supporting different traits than the existing
Arbitrary
.The quickcheck crate is similar and may be less complex to understand than the proptest crate.
The text was updated successfully, but these errors were encountered: