Is it possible to migrate a java or python benchmark from OSS-Fuzz to fuzzbench?

[FankeTong]

I am hoping to measure some java or python benchmarks with fuzzbench. So I migrate a java library (java-xmlbuilder) from OSS-Fuzz by copying Dockerfile, build.sh and XmlBuilderFuzzer.java. Then I try to test it on fuzzbench using the following steps:

make -j builder-libfuzzer-java-xmlbuilder_xmlbuilderfuzzer 
make -j run-libfuzzer-java-xmlbuilder_xmlbuilderfuzzer 

But an error occured at final step:

docker run \
--cpus=1 \
--shm-size=2g \
--cap-add SYS_NICE \
--cap-add SYS_PTRACE \
-e FUZZ_OUTSIDE_EXPERIMENT=1 \
-e FORCE_LOCAL=1 \
-e TRIAL_ID=1 \
-e FUZZER=libfuzzer \
-e BENCHMARK=java-xmlbuilder_xmlbuilderfuzzer \
-e FUZZ_TARGET=XmlBuilderFuzzer \
-it gcr.io/fuzzbench/runners/libfuzzer/java-xmlbuilder_xmlbuilderfuzzer
INFO:root:Doing trial as part of experiment.
INFO:root:Starting trial.
WARNING:root:max_total_time is None. Fuzzing indefinitely.
[run_fuzzer] Running command: /out/XmlBuilderFuzzer -print_final_stats=1 -close_fd_mask=3 -fork=1 -ignore_ooms=1 -ignore_timeouts=1 -ignore_crashes=1 -entropic=1 -keep_seed=1 -cross_over_uniform_dist=1 -entropic_scale_per_exec_time=1 -detect_leaks=0 -artifact_prefix=/out/corpus/crashes/ -dict=/out/XmlBuilderFuzzer.dict /out/corpus/corpus /out/seeds
/out/XmlBuilderFuzzer: line 12: /out/jazzer_driver: No such file or directory
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/src/fuzzers/libfuzzer/fuzzer.py", line 41, in fuzz
    run_fuzzer(input_corpus, output_corpus, target_binary)
  File "/src/fuzzers/libfuzzer/fuzzer.py", line 102, in run_fuzzer
    subprocess.check_call(command)
  File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/out/XmlBuilderFuzzer', '-print_final_stats=1', '-close_fd_mask=3', '-fork=1', '-ignore_ooms=1', '-ignore_timeouts=1', '-ignore_crashes=1', '-entropic=1', '-keep_seed=1', '-cross_over_uniform_dist=1', '-entropic_scale_per_exec_time=1', '-detect_leaks=0', '-artifact_prefix=/out/corpus/crashes/', '-dict=/out/XmlBuilderFuzzer.dict', '/out/corpus/corpus', '/out/seeds']' returned non-zero exit status 127.
ERROR:root:Executed command: "nice -n 5 python3 -u -c from fuzzers.libfuzzer import fuzzer; fuzzer.fuzz("/out/seeds", "/out/corpus", "/out/XmlBuilderFuzzer")" returned: 1.
ERROR:root:Fuzz process returned nonzero. Extras: {'traceback': 'Traceback (most recent call last):\n  File "/src/experiment/runner.py", line 200, in run_fuzzer\n    new_process.execute(command,\n  File "/src/common/new_process.py", line 125, in execute\n    raise subprocess.CalledProcessError(retcode, command)\nsubprocess.CalledProcessError: Command \'[\'nice\', \'-n\', \'5\', \'python3\', \'-u\', \'-c\', \'from fuzzers.libfuzzer import fuzzer; fuzzer.fuzz("/out/seeds", "/out/corpus", "/out/XmlBuilderFuzzer")\']\' returned non-zero exit status 1.\n'}
INFO:root:Doing final sync.
make: *** [docker/generated.mk:275779: run-libfuzzer-java-xmlbuilder_xmlbuilderfuzzer] Error 1

Maybe the missing jazzer_driver is the key to this problem? When I test java-xmlbuilder on OSS-Fuzz, it works fine and the jazzer_fuzzer is in the /out directory. checking the /out directory of both images，the benchmark-image of fuzzbench is missing everything of jazzer:

# OSS-Fuzz /out
XmlBuilderFuzzer        XmlBuilderFuzzer.dict                   java-xmlbuilder.jar      jazzer_driver                 llvm-symbolizer
XmlBuilderFuzzer.class  XmlBuilderFuzzer_libfuzzer_address_out  jazzer_agent_deploy.jar  jazzer_driver_with_sanitizer
# fuzzbench /out
XmlBuilderFuzzer  XmlBuilderFuzzer.class  XmlBuilderFuzzer.dict  java-xmlbuilder.jar

I am not familiar with the image building process of oss-fuzz and fuzzbench. So I assuming the difference of it causing this problem. Is there any way to support java and python benchmark on fuzzbench? Any help would be appreciated...

The composite of the added benchmark java-xmlbuilder_xmlbuilderfuzzer as follow

Dockfile

FROM gcr.io/oss-fuzz-base/base-builder-jvm
RUN sed -i s@/archive.ubuntu.com/@/mirrors.aliyun.com/@g /etc/apt/sources.list
RUN curl -L https://archive.apache.org/dist/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.zip \
      -o maven.zip && \
      unzip maven.zip -d $SRC/maven && \
      rm maven.zip
ENV MVN $SRC/maven/apache-maven-3.6.3/bin/mvn
RUN git clone --depth 1 https://github.com/google/AFL afl
RUN git clone --depth 1 https://github.com/jmurty/java-xmlbuilder java-xmlbuilder
COPY build.sh *.java $SRC/
WORKDIR $SRC/java-xmlbuilder

build.sh

$MVN clean package -Dmaven.javadoc.skip=true -DskipTests=true -Dpmd.skip=true \
    -Dencoding=UTF-8 -Dmaven.antrun.skip=true -Dcheckstyle.skip=true \
    -Denforcer.fail=false org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade
CURRENT_VERSION=$($MVN org.apache.maven.plugins:maven-help-plugin:3.2.0:evaluate \
 -Dexpression=project.version -q -DforceStdout)

cp "target/java-xmlbuilder-$CURRENT_VERSION.jar" $OUT/java-xmlbuilder.jar

ALL_JARS="java-xmlbuilder.jar"

# The classpath at build-time includes the project jars in $OUT as well as the
# Jazzer API.
BUILD_CLASSPATH=$(echo $ALL_JARS | xargs printf -- "$OUT/%s:"):$JAZZER_API_PATH

# All .jar and .class files lie in the same directory as the fuzzer at runtime.
RUNTIME_CLASSPATH=$(echo $ALL_JARS | xargs printf -- "\$this_dir/%s:"):\$this_dir

for fuzzer in $(find $SRC -name '*Fuzzer.java')
do
  fuzzer_basename=$(basename -s .java $fuzzer)
  javac -cp $BUILD_CLASSPATH $fuzzer
  cp $SRC/$fuzzer_basename.class $OUT/

  # Create an execution wrapper that executes Jazzer with the correct arguments.
  echo "#!/bin/bash

  # LLVMFuzzerTestOneInput for fuzzer detection.
  this_dir=\$(dirname "\$0")
  if [[ "\$@" =~ (^| )-runs=[0-9]+($| ) ]]
  then
    mem_settings='-Xmx1900m:-Xss900k'
  else
    mem_settings='-Xmx2048m:-Xss1024k'
  fi

  LD_LIBRARY_PATH="$JVM_LD_LIBRARY_PATH":\$this_dir \
    \$this_dir/jazzer_driver                        \
    --agent_path=\$this_dir/jazzer_agent_deploy.jar \
    --cp=$RUNTIME_CLASSPATH                         \
    --target_class=$fuzzer_basename                 \
    --jvm_args="\$mem_settings"                     \
    \$@" > $OUT/$fuzzer_basename

    chmod u+x $OUT/$fuzzer_basename
done

cp $SRC/afl/dictionaries/xml.dict $OUT/XmlBuilderFuzzer.dict

benchmark.yaml

fuzz_target: XmlBuilderFuzzer
project: java-xmlbuilder

XmlBuilderFuzzer.java

import com.code_intelligence.jazzer.api.FuzzedDataProvider;
import com.jamesmurty.utils.XMLBuilder;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.xpath.XPathExpressionException;
import org.w3c.dom.DOMException;

public class XmlBuilderFuzzer {
  public static void fuzzerTestOneInput(FuzzedDataProvider data) {
    try {
      int[] choices = data.consumeInts(data.consumeInt(1, 10));
      XMLBuilder builder = XMLBuilder.create(data.consumeString(data.remainingBytes() / 2));

      for (Integer choice : choices) {
        switch (choice % 13) {
          case 0:
            builder = builder.stripWhitespaceOnlyTextNodes();
            break;
          case 1:
            builder = builder.up(data.consumeInt());
            break;
          case 2:
            builder = builder.elem(data.consumeRemainingAsString());
            break;
          case 3:
            builder = builder.elementBefore(data.consumeRemainingAsString());
            break;
          case 4:
            builder = builder.attr(
                data.consumeString(data.remainingBytes() / 2), data.consumeRemainingAsString());
            break;
          case 5:
            builder = builder.text(data.consumeRemainingAsString());
            break;
          case 6:
            builder = builder.data(data.consumeRemainingAsString());
            break;
          case 7:
            builder = builder.data(data.consumeRemainingAsBytes());
            break;
          case 8:
            builder = builder.cmnt(data.consumeRemainingAsString());
            break;
          case 9:
            builder = builder.inst(
                data.consumeString(data.remainingBytes() / 2), data.consumeRemainingAsString());
            break;
          case 10:
            builder = builder.insertInstruction(
                data.consumeString(data.remainingBytes() / 2), data.consumeRemainingAsString());
            break;
          case 11:
            builder = builder.ref(data.consumeRemainingAsString());
            break;
          case 12:
            builder = builder.ns(data.consumeRemainingAsString());
            break;
        }
      }
    } catch (ParserConfigurationException | XPathExpressionException | DOMException
        | IllegalStateException e) {
      // Known exception
    }
  }
}

[jonathanmetzman]

It will be very hard.
Most fuzzers in FuzzBench don't support Java.
And fuzzbench relies on clang coverage which only works for native code.