Skip to content

Latest commit

 

History

History
104 lines (79 loc) · 10.1 KB

README.md

File metadata and controls

104 lines (79 loc) · 10.1 KB
Raw

SJFuzz

Seed & Mutator Scheduling for JVM Fuzzing

JVM Versions

JVM Versions
OpenJDK 8u232, 8u275, 9.0.4, 11.0.5, 11.0.9
OpenJ9 8u232, 8u275, 9.0.4, 11.0.5, 11.0.9
OracleJDK 6, 7, 8
DragonWell 8u232, 8u275, 9.0.4, 11.0.5, 11.0.9

Issues Found By SJFuzz

Issues Impact Versions Confirmed Link
J9 (8u275) crashed while J9 (9.0.4) and J9 (11.0.9) did not. J9 (8u275) J9 (8u275) eclipse-openj9/openj9#11683
J9 failed to pass the JUnit test after throwing StackOverflowError for a previous test. When removing that "previous test", J9 pass the test. J9: 8u232, 9.0.4, 11.0.5 J9: 8u232, 9.0.4, 11.0.5 eclipse-openj9/openj9#9381
J9 (8u275) threw an AssertionError; J9 (11.0.9) threw an ArrayIndexOutOfBoundsException; OpenJDK (8u275, 9.0.4, 11.0.9) and J9 (9.0.4) threw an VerifyError; J9 (8u275) and J9 (11.0.9) J9 (8u275) and J9 (11.0.9) eclipse-openj9/openj9#11685
J9 (8u275, 11.0.9) passed the JUnit test while OpenJDK (8u275, 9.0.4, 11.0.9) and J9 (9.0.4) threw a VerifyError. J9 (8u275, 11.0.9) J9 (8u275, 11.0.9) eclipse-openj9/openj9#11684
J9: failed to throw VerifyError? J9(8u232) J9(11.0.5) J9(8u232) J9(11.0.5) eclipse-openj9/openj9#9385
OpenJDK 8u232, 9.0.4 executed a different program path compared to OpenJDK11, J9 OpenJDK 8u232, 9.0.4 https://bugs.openjdk.java.net/browse/JDK-8259961
J9 (8u275) and J9 (11.0.9) failed to threw AssertionError J9 (8u275) J9 (11.0.9) J9 (8u275) J9 (11.0.9) eclipse-openj9/openj9#11721
J9 (11.0.9) crashed while executing a test file on a heavy-load server. J9 (11.0.9) J9 (11.0.9) eclipse-openj9/openj9#11725
JDK can not find MANIFEST.MF files by ClassLoader.getResources() OracleJDK6, OracleJDK7, OracleJDK8, OpenJDK8u232 OracleJDK6, OracleJDK7, OracleJDK8, OpenJDK8u232 https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8244083
J9 failed to execute all JUnit tests in a test file (caused by initializationError) J9: 8u232, 9.0.4, 11.0.5 J9: 8u232, 9.0.4, 11.0.5 eclipse-openj9/openj9#9357
In JUnit tests, J9 said can not find the definition of class which is actually found in previous tests? J9: 8u232, 9.0.4, 11.0.5 same issue as in 9357 eclipse-openj9/openj9#9336
J9 failed to detect that the class is not defined during the loading stage? J9: 8u232, 9.0.4, 11.0.5 same issue as in 9357 eclipse-openj9/openj9#9323
Dragonwell (8u275): getMessage() for an ArrayIndexOutOfBoundsException instance is null Dragonwell (8u275) dragonwell-project/dragonwell8#186
J9 (9.0.4) threw NullPointerException while other JVMs threw AssertionError. J9 (9.0.4) eclipse-openj9/openj9#11723
Dragonwell8u232: ClassLoader.getResources() can not work properly Dragonwell8u232 dragonwell-project/dragonwell8#99
OpenJDK (8u232, 9.0.4, 11.0.5) skipped an erroneous instruction OpenJDK (8u232, 9.0.4, 11.0.5) https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8243483
A possible bug in Verification: Dragonwell skipped an erroneous instruction? Dragonwell8u232 Dragonwell11.0.5 dragonwell-project/dragonwell8#97
J9 failed to throw the IllegalMonitorStateException J9: 8u232, 9.0.4, 11.0.5 eclipse-openj9/openj9#9276
J9 Verification Issue: did not check uninitialized register? J9 (8u232) eclipse-openj9/openj9#9271
A possible bug found in JVM differential testing dw Dragonwell8u232 dragonwell-project/dragonwell8#89
OpenJDK did not report ClassFormatError when class name is illegal OpenJDK: 8u232 8u242 https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8242352
OpenJDK 8u275 crashed on a heavy-load server. OpenJDK 8u275 https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8262342
OpenJDK 9.0.4 crashed on a heavy-load server. OpenJDK 9.0.4 https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8262343
OpenJDK 11.0.9 crashed on a heavy-load server. OpenJDK 11.0.9 http://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8262344
Total 46 20

Performance under different configurations

In general, the data within the same cluster indicate that they are more similar to one another than the ones within other clusters. Therefore, the more clusters we have, the more diverse the overall data set is. Specifically, we adapt the widely-used DBScanClustering, a density-based clustering algorithm, to evaluate the diversity of the generated class files in terms of the generated cluster number. DBScanClustering adopts two key parameters, radius and minPts, which are both set to 1 following the studied settings from prior work.

To evaluate the impact of the parameter settings on SJFuzz, we explore different explorationRate (in Algorithm 2) and bound (in Algorithm 1) values on all our benchmarks. We show their impacts on the DBScanClustering-based diversity of the generated class files as in the following box plots. In particular, we set bound to the default 20 and investigate the impact of different explorationRates, i.e., 0.05, 0.1, 0.3 and 0.5 (left sub-figure); we also set explorationRate to the default 0.1 and investigate the impact of different bounds (right sub-figure), i.e., 10, 20, 50 and 100. In the figure, each box plot presents the diversity distribution for one configuration across all our studied subjects. We can observe that different configurations do not impact our approach much and all configurations substantially outperform classming (the gray box plots), indicating the effectiveness and stability of SJFuzz.

boxplot

Also, we tried different distance algorithms other than the Levenshtein distance for SJFuzz. As shown in the following figure, different distance algorithms do not impact the effectiveness of the SJFuzz much.

boxplot

Usage

Main-entry Mode

In the main method of DjfuzzFrameworkResumable.java, add following code to start test generation:

fwk.process(fully_qualified_class_name, iteration_number,
            parameters_for_the_class, class_path,
            dependencies_path, jvm_options);

For example:

fwk.process("org.sunflow.Benchmark", 50000,
            new String[]{"-bench","2","256"},
            "./sootOutput/sunflow-0.07.2/",
            "dependencies/janino-2.5.15.jar", "");

JUnit-entry Mode

In the main method of DjfuzzFrameworkResumableAdvanced.java, add following code to start test generation:

Main.useJunit(junit_jar_path, hamcrest_jar_path,
            tool_jar_path, fully_qualified_name_of_test_class);
fwk.process(fully_qualified_class_name, iteration_number,
            parameters_for_the _class, class_path,
            dependencies_path, jvm_options);

For example:

Main.useJunit("./junit-4.12.jar", "./hamcrest-core-1.3.jar",
            "./tools.jar", "org.apache.tools.ant.AntClassLoaderTest");
fwk.process("org.apache.tools.ant.AntClassLoader", 10000, args,
            "./sootOutput/junit-ant/",
            "", "");

Collect Generated Classes

Generated live mutant is in directory ./AcceptHistory/ and ./RejectHistory/. Mutants with no live code is in ./nolivecode/. All the generated mutants have backup file in ./tmp/. The path of the discrepancy-inducing class file will be written into differential-log.txt.