Reorganize shootout benchmarks into a single directory

.github/workflows/benchmarks_native.yaml

@@ -7,14 +7,14 @@ on:
     branches: [ main ]
 
 jobs:
-
-  Build_Native_X86-64:
+  native:
+    name: Build and run native benchmarks
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
     - name: Install nightly
       uses: actions-rs/toolchain@v1
       with:
         toolchain: nightly
-    - name: Rebuild benchmarks docker
-      run: benchmarks/build-all-native.sh --run
+    - name: Rebuild and run native benchmarks using Docker
+      run: benchmarks/build-all-native.sh --run

benchmarks/.gitignore

@@ -1,2 +1,2 @@
-*/benchmark.so
+*/*.so
 */target

benchmarks/README.md

@@ -71,53 +71,30 @@ through the `validate` command:
 $ cargo run -- validate path/to/benchmark.wasm
 ```
 
+## Compatibility Requirements for Native Execution
 
-## Compatibility Requirements for the Native Engine
-
-In addition to knowing the performance of the benchmark when targeting Wasm,
-a user may also want to know the performance of that same benchmark when native
-is the original target. To facilitate this, Sightglass includes a native
-engine (similar to the Wasm engine) that can execute a natively compiled program
-such that the same high level source can be used to compile both Wasm and native
-targets without change. Because it is intended that the same high-level source be used,
-requirements listed in the section "Minimal Technical Requirement" above
-are required. In addition though, the benchmark folder is required to supply a Cargo based
-build support that targets a benchmark.so file copied to the base of the benchmark's
-target directory. Specifically:
-
-* A Cargo.toml must be included and the base of the benchmark directory, <benchmark>/Cargo.toml,
-such that "cargo run" produces a benchmark.so dynamically linked library file at the base of the
-target folder, <benchmark>/target/benchmark.so
-
-* The "main" function must be renamed to "native_entry". For C and C++ based source this can be done
-with a simple define directive passed to CC. For example to compile a shootout benchmark the following
-command is used:
-
-
- ```
- let output = Command::new("cc")
-        .args([
-            "-O3",
-            "-Dmain=native_entry",
-            "-fPIC",
-            "-I.",
-            "-shared",
-            "-o",
-            "./target/benchmark.so",
-            "benchmark.c",
-        ])
-        .output()
-        .expect("failed to compile native benchmark");
-    io::stdout().write_all(&output.stdout).unwrap();
-    io::stderr().write_all(&output.stderr).unwrap();
-````
-
-* Also, a symbolic link should be included to Dockerfile.native to allow building in a container. Because this Dockerfile
-will simply copy the directory context and call cargo run as if compiling directly on host, there should be no need to
-supply a custom Dockerfile to make a container based build work.
-
-* Support for native is optional. If a benchmark is added for Wasm, corresponding build support for native is optional and
-will not break CI if it is not included.
+Sightglass can also measure the performance of a subset of benchmarks compiled
+to native code (i.e., not WebAssembly). To compile these benchmarks without
+changing their source code, this involves a delicate interface with the [native
+engine] with some additional requirements beyond the [Minimal Technical
+Requirements] noted above:
+
+[native engine]: ../engines/native
+[Minimal Technical Requirements]: #minimal-technical-requirements
+
+* Generate an ELF shared library linked to the [native engine] shared library to
+  provide definitions for `bench_start` and `bench_end`.
+
+* Rename the `main` function to `native_entry`. For C- and C++-based source this
+  can be done with a simple define directive passed to `cc` (e.g.,
+  `-Dmain=native_entry`).
+
+* Provide reproducible builds via a `Dockerfile.native` file (see
+  [`build-native.sh`](./build-native.sh)).
+
+Note that support for native execution is optional: adding a WebAssembly
+benchmark does not imply the need to support its native equivalent &mdash; CI
+will not fail if it is not included.
 
 ## Additional Requirements
 

benchmarks/all.suite

@@ -94,23 +94,23 @@ meshoptimizer/benchmark.wasm
 noop/benchmark.wasm
 pulldown-cmark/benchmark.wasm
 regex/benchmark.wasm
-shootout-ackermann/benchmark.wasm
-shootout-base64/benchmark.wasm
-shootout-ctype/benchmark.wasm
-shootout-ed25519/benchmark.wasm
-shootout-fib2/benchmark.wasm
-shootout-gimli/benchmark.wasm
-shootout-heapsort/benchmark.wasm
-shootout-keccak/benchmark.wasm
-shootout-matrix/benchmark.wasm
-shootout-memmove/benchmark.wasm
-shootout-minicsv/benchmark.wasm
-shootout-nestedloop/benchmark.wasm
-shootout-random/benchmark.wasm
-shootout-ratelimit/benchmark.wasm
-shootout-seqhash/benchmark.wasm
-shootout-sieve/benchmark.wasm
-shootout-switch/benchmark.wasm
-shootout-xblabla20/benchmark.wasm
-shootout-xchacha20/benchmark.wasm
+shootout/shootout-ackermann.wasm
+shootout/shootout-base64.wasm
+shootout/shootout-ctype.wasm
+shootout/shootout-ed25519.wasm
+shootout/shootout-fib2.wasm
+shootout/shootout-gimli.wasm
+shootout/shootout-heapsort.wasm
+shootout/shootout-keccak.wasm
+shootout/shootout-matrix.wasm
+shootout/shootout-memmove.wasm
+shootout/shootout-minicsv.wasm
+shootout/shootout-nestedloop.wasm
+shootout/shootout-random.wasm
+shootout/shootout-ratelimit.wasm
+shootout/shootout-seqhash.wasm
+shootout/shootout-sieve.wasm
+shootout/shootout-switch.wasm
+shootout/shootout-xblabla20.wasm
+shootout/shootout-xchacha20.wasm
 spidermonkey/benchmark.wasm

benchmarks/build-all-native.sh

@@ -2,9 +2,11 @@
 
 # Either build all of the benchmark's native target or a random subset of them.
 #
-# Usage: ./build-all-native.sh <number of benchmarks>
-# - <number of benchmarks>, an optional number of benchmarks to build; if provided, this script will
-#   randomize the list of benchmarks and pick a subset of them to build
+# Usage: ./build-all-native.sh <number of benchmarks> [--run]
+# - <number of benchmarks>, an optional number of benchmarks to build; if
+#   provided, this script will randomize the list of benchmarks and pick a
+#   subset of them to build
+# - --run, if set, this script will attempt to run the built benchmarks
 
 set -e
 
@@ -26,6 +28,8 @@ for DOCKERFILE in $DOCKERFILES; do
     BENCHMARK_DIR=$(dirname $DOCKERFILE)
     $BUILD_SCRIPT $BENCHMARK_DIR
     if [[ $* == *$FLAG* ]]; then
-        $RUN_SCRIPT $BENCHMARK_DIR/target/benchmark.so
+        for BENCHMARK in $(find $BENCHMARK_DIR -name '*.so'); do
+            $RUN_SCRIPT $BENCHMARK
+        done
     fi
 done

benchmarks/build-native.sh

@@ -1,35 +1,27 @@
 #!/usr/bin/env bash
 
-# Build a single native benchmark
+# Build a single native benchmark.
+#
+# Usage: ./build-native.sh [--host] <path to benchmark directory>
 
 set -e
 
 BENCHMARKS_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 SIGHTGLASS_BASE=$(dirname $BENCHMARKS_DIR)
 ENGINE=$SIGHTGLASS_BASE/engines/native/libengine.so
-
-BENCHMARK_DIR="";
-
-for ARG in "$@"; do
-    if [[ -d $ARG ]]; then
-        BENCHMARK_DIR=$ARG;
-        break
-    fi
-done
+BENCHMARK_DIR=$1;
 
 print_header() {
     >&2 echo
     >&2 echo ===== $@ =====
 }
 
-
-if [[ $BENCHMARK_DIR == "" ]]; then
+if [[ ! -d $BENCHMARK_DIR ]]; then
     echo "Unknown benchmark directory; usage: ./build-native.sh [--host] <path to benchmark directory>"
     exit 1
 fi
 
-
-# Build engine if not availble since it needs to be present for linking
+# Build engine if not availble since it needs to be present for linking.
 if [[ ! -f $ENGINE ]]; then
     cd $SIGHTGLASS_BASE/engines/native/libengine/
     cargo build --release
@@ -40,29 +32,26 @@ fi
 # Build directly on host
 FLAG='--host'
 if [[ $* == *$FLAG* ]]; then
-
-print_header "Build Native Benchmark Using Host"
-(set -x; cd $BENCHMARK_DIR && cargo run --release)
-
-
+    print_header "Build native benchmark directly on host (no Docker)"
+    (set -x; cd $BENCHMARK_DIR && ./build-native.sh)
 else
-
-BENCHMARK_NAME=$(readlink -f $BENCHMARK_DIR | xargs basename)
-IMAGE_NAME=sightglass-benchmark-$BENCHMARK_NAME-native
-
-# To allow the use of symlinks in the benchmark directories (docker ignores them), we `tar` up the
-# directory and `--dereference` (i.e., follow) all symlinks provided.
-print_header "Create build context"
-TMP_TAR=$(mktemp /tmp/sightglass-benchmark-dir-XXXXXX.tar)
-(set -x; cd $BENCHMARK_DIR && ln -f -s ../../engines/native/libengine.so ./libengine.so && tar --create --file $TMP_TAR --dereference --verbose . && rm libengine.so)
-
-# Build the benchmark and extract the build results from the container
-print_header "Build Native Benchmark Using Host"
-(set -x; docker build -f Dockerfile.native --tag $IMAGE_NAME - < $TMP_TAR)
-CONTAINER_ID=$(set -x; docker create $IMAGE_NAME)
-(set -x; mkdir -p $BENCHMARK_DIR/target; docker cp --follow-link $CONTAINER_ID:/benchmark/target/. $BENCHMARK_DIR/target/;)
-
-# Copy host files to container and build inside a container
-
+    BENCHMARK_NAME=$(readlink -f $BENCHMARK_DIR | xargs basename)
+    IMAGE_NAME=sightglass-benchmark-$BENCHMARK_NAME-native
+
+    # To allow the use of symlinks in the benchmark directories (docker ignores
+    # them), we `tar` up the directory and `--dereference` (i.e., follow) all
+    # symlinks provided.
+    print_header "Create build context"
+    TMP_TAR=$(mktemp /tmp/sightglass-benchmark-dir-XXXXXX.tar)
+    (set -x; cd $BENCHMARK_DIR && \
+        ln -f -s $ENGINE ./libengine.so && \
+        tar --create --file $TMP_TAR --dereference --verbose . && \
+        rm libengine.so)
+
+    # Build the benchmark and extract the build results from the container.
+    print_header "Build native benchmark in Docker"
+    (set -x; docker build -f Dockerfile.native --tag $IMAGE_NAME - < $TMP_TAR)
+    CONTAINER_ID=$(set -x; docker create $IMAGE_NAME)
+    (set -x; docker cp --follow-link $CONTAINER_ID:/benchmark/. $BENCHMARK_DIR/;)
 fi
 

benchmarks/run-native.sh

@@ -1,40 +1,22 @@
 #!/usr/bin/env bash
 
-# Run a single native benchmark that is already built
+# Run a single native benchmark that is already built.
 #
-# Usage: ./run-native.sh <path-to-benchmark-folder/target/benchmark.so>
+# Usage: ./run-native.sh <path-to-benchmark-folder/benchmark.so>
 
 set -e
 
 BENCHMARKS_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 SIGHTGLASS_BASE=$(dirname $BENCHMARKS_DIR)
-
-SIGHTGLASS="cargo +nightly run --release --bin sightglass-cli --"
-
+SIGHTGLASS="cargo run --release --bin sightglass-cli --"
 ENGINE=$SIGHTGLASS_BASE/engines/native/libengine.so
-
-BENCHMARK_NATIVE_SO="";
-
-for ARG in "$@"; do
-    if [[ -f $ARG ]]; then
-        BENCHMARK_NATIVE_SO=$ARG;
-        break
-    fi
-done
+BENCHMARK_NATIVE_SO=$1
 
 if [[ -z $BENCHMARK_NATIVE_SO ]]; then
-    echo "Missing benchmark"
-    echo "Usage: run-native.sh <path-to-benchmark-folder/target/benchmark.so>"
-    exit
+    echo "Usage: run-native.sh <path-to-benchmark-folder/benchmark.so>"
+    exit 1
 fi
 
-BENCHMARK_NATIVE_SO="$(realpath $BENCHMARK_NATIVE_SO)"
-
-print_header() {
-    >&2 echo
-    >&2 echo ===== $@ =====
-}
-
 # If an engine is not available, build it.
 if [[ ! -f $ENGINE ]]; then
     cd $SIGHTGLASS_BASE/engines/native/libengine/
@@ -43,10 +25,24 @@ if [[ ! -f $ENGINE ]]; then
     cd - > /dev/null
 fi
 
-# Run a benchmark with the newly created library
+# Because of some hard-coding in the native engine (TODO:
+# https://github.com/bytecodealliance/sightglass/issues/259), we need to set up
+# the temporary directory with the hard-coded paths.
+BENCHMARK_NATIVE_SO="$(realpath $BENCHMARK_NATIVE_SO)"
+MD5SUM=$(md5sum $BENCHMARK_NATIVE_SO | awk '{ print $1 }')
+TMP_BENCHMARK_DIR=/tmp/sightglass-benchmark-native-$MD5SUM
+mkdir -p $TMP_BENCHMARK_DIR
+(set -x; ln -fs $BENCHMARK_NATIVE_SO $TMP_BENCHMARK_DIR/benchmark.so)
+BENCHMARK_DIR=$(dirname $BENCHMARK_NATIVE_SO)
+NAME=$(basename $BENCHMARK_NATIVE_SO .so);
+for FILE in $(find $BENCHMARK_DIR -name "$NAME*.input"); do
+    (set -x; ln -fs $FILE $TMP_BENCHMARK_DIR/)
+done
+
+# Run a benchmark with the native library.
 cd $SIGHTGLASS_BASE
-BENCH_DIR=$(dirname $BENCHMARK_NATIVE_SO)
-echo $BENCH_DIR
-LD_LIBRARY_PATH=./engines/native/ $SIGHTGLASS benchmark --engine engines/native/libengine.so --working-dir $BENCH_DIR -- $BENCHMARK_NATIVE_SO
+ENGINE_DIR=$(dirname $ENGINE)
+(set -x; LD_LIBRARY_PATH=$ENGINE_DIR $SIGHTGLASS benchmark --engine $ENGINE \
+    --processes 1 --iterations-per-process 3 \
+    --working-dir $TMP_BENCHMARK_DIR -- $TMP_BENCHMARK_DIR/benchmark.so)
 cd - > /dev/null
-