Parameterizing at build time

[GAJaloyan]

Hi,

I'm trying to use JQF for fuzzing a java method. It works well, but I want to make the number of trials parameterized at build time (or through an environment variable). Is there an easy way to do this? (as of now, I'm using bash hooks in my gradle build configuration to do macro expansion at compile time)

@Property(trials = 10_000)
public void testEquivalence(
	       @Probability(....))

Also, is there a solution to run those 10000 trials on N cores (typically in the order of 256)? I've seen that junit is able to run different tests in parallel, but it seems that a fuzzing task is considered as a single test only.

Thanks, George.

[rohanpadhye]

The @Property annotation is used for vanilla JUnit-Quickcheck tests, and it is not controlled by JQF. You can ask on https://github.com/pholser/junit-quickcheck/issues if this can be controlled at run-time.

If you are using the @Fuzz annotation provided by JQF, then the test can be executed in one of two ways:

1. By explicitly running mvn jqf:fuzz -Dclass=<..> -Dmethod=<...> as a command, where you can optionally give a timeout using -Dtime=60s or similar. In this case, JQF will instrument the bytecode to do coverage-guided fuzzing.
2. By running the test as part of your normal test suite, such as mvn test. JUnit will pick up any @Fuzz annotated test driver and run it as a random test without coverage feedback (because the code is not instrumented when run this way), similar to what quickcheck would have done with the @Property annotation. In this instance, the default is to run the test for 100 executions of random input, but you can override that by setting the system property jqf.quickcheck.trials. For example, mvn test -Djqf.quickcheck.trials=1234 or set it in the pom.xml configuration using environment variables.

The fuzzing task does indeed run single threaded at this time. JQF was mainly designed for people running the fuzzer via something like mvn jqf:fuzz. With this command, you can use a bash script to run the fuzzer on 256 different cores, but each independent run will use its own internal state for things like coverage feedback. There is a way to run fuzzers in parallel and share coverage info via files on disk, but this logic is not currently in the release version. When @Fuzz tests are run with the default Maven Surefire plugin with mvn test, they always run single threaded because this mode is just a wrapper around a simple while loop. See NoGuidance.