chore: Cl/ci3.2 (#10919)

Further iteration towards full CI3. TLDR: Working towards ~10m repo test time. * Begin to separate out "building of tests" (mainly thinking of C++ and Rust). We don't want to do this on a fast bootstrap, but we do want to do it if we're going to run the tests. And moving towards the new testing model we need to separate building and running of tests. * Introduce `test-cmds` cmd on bootstrap scripts. Returns a list of commands, that if run from repo root, execute individual (usually) tests. * Note this also introduces the standard of `./scripts/run_test.sh` being a script that given some succinct arguments, can run a single test. * Introduce `test-all` (eventually to become just `test`) in root bootstrap.sh. No args runs all tests, or you can give it a list of folders to projects with their own bootstrap scripts and it'll run their tests. Runs in 10m20s. Currently skipping some things (see TODO below). Reports slow tests after run. * Note this also runs our TS project tests *directly as javascript*. i.e. it's assumed the tests have all been compiled to the dest folder and have whatever they need to operate. Hitting yarn + transpiler is just gruesome use of resources. * Improve cache script to not deal with env vars, but just args. If the args is a file, its treated as a rebuild patterns file, otherwise treated as a pattern itself. * Remove `TEST=0/1` flag. Unnecessary. Normal bootstraps don't run tests, and If i request to run tests I want them to run. So the "skip tests if cache flag exists" only needs to be applied if `CI=1`. * Get's rid of all hardcoded srs paths in favour of making function call to get the path. Will check environment variables first, and fallback on hardcoded path (now in one place). I ultimately didn't need this like I thought I would, but it's the right move anyway, and will make the switch to the flat crs easier. * Bit of refactoring to remove "right drift" of cache blocks. i.e. return if nothing to do instead of enclosing in an if statement. * bb.js uses @swc/jest like yarn-projects does. * Delete `bootstrap` folder. Is was there to help test the bootstrap script in CI, but now we use the bootstrap script in CI. * Add build cache to `boxes`. * Enable extended globs in CI3 scripts. * Revert back to default jest reporter, unless running all tests from root, then it uses summary reporter. TODO: - [ ] kv-store tests - [x] TXE for contracts/aztec.nr tests - [x] noir js packages tests - [ ] Skipping tests matching `test_caches_open|requests` in noir tests. - [x] Standardise how tests are skipped so we can see in one place. --------- Co-authored-by: ludamad <[email protected]>
AztecProtocol · Jan 2, 2025 · 49dacc3 · 49dacc3
1 parent cd59f2e
commit 49dacc3
Show file tree

Hide file tree

Showing 187 changed files with 1,037 additions and 975 deletions.
diff --git a/barretenberg/acir_tests/bootstrap.sh b/barretenberg/acir_tests/bootstrap.sh
@@ -4,158 +4,158 @@ source $(git rev-parse --show-toplevel)/ci3/source_bootstrap
 cmd=${1:-}
 export CRS_PATH=$HOME/.bb-crs
 
-function build {
+function build_tests {
   set -eu
-  if [ ! -d acir_tests ]; then
-    cp -R ../../noir/noir-repo/test_programs/execution_success acir_tests
-    # Running these requires extra gluecode so they're skipped.
-    rm -rf acir_tests/{diamond_deps_0,workspace,workspace_default_member}
-    # TODO(https://github.com/AztecProtocol/barretenberg/issues/1108): problem regardless the proof system used
-    rm -rf acir_tests/regression_5045
-  fi
+
+  github_group "acir_tests build"
+
+  rm -rf acir_tests
+  cp -R ../../noir/noir-repo/test_programs/execution_success acir_tests
+  # Running these requires extra gluecode so they're skipped.
+  rm -rf acir_tests/{diamond_deps_0,workspace,workspace_default_member}
+  # TODO(https://github.com/AztecProtocol/barretenberg/issues/1108): problem regardless the proof system used
+  rm -rf acir_tests/regression_5045
 
   # COMPILE=2 only compiles the test.
-  github_group "acir_tests compiling"
-  parallel --joblog joblog.txt --line-buffered 'COMPILE=2 ./run_test.sh $(basename {})' ::: ./acir_tests/*
-  github_endgroup
+  denoise "parallel --joblog joblog.txt --line-buffered 'COMPILE=2 ./run_test.sh \$(basename {})' ::: ./acir_tests/*"
 
   # TODO: This actually breaks things, but shouldn't. We want to do it here and not maintain manually.
   # Regenerate verify_honk_proof recursive input.
   # local bb=$(realpath ../cpp/build/bin/bb)
   # (cd ./acir_tests/assert_statement && \
   #   $bb write_recursion_inputs_honk -b ./target/program.json -o ../verify_honk_proof --recursive)
 
-  github_group "acir_tests updating yarn"
   # Update yarn.lock so it can be committed.
   # Be lenient about bb.js hash changing, even if we try to minimize the occurrences.
-  (cd browser-test-app && yarn add --dev @aztec/bb.js@../../ts && yarn)
-  (cd headless-test && yarn)
-  (cd sol-test && yarn)
+  denoise "cd browser-test-app && yarn add --dev @aztec/bb.js@../../ts && yarn"
+  denoise "cd headless-test && yarn"
+  denoise "cd sol-test && yarn"
   # The md5sum of everything is the same after each yarn call.
   # Yet seemingly yarn's content hash will churn unless we reset timestamps
   find {headless-test,browser-test-app} -exec touch -t 197001010000 {} + 2>/dev/null || true
-  github_endgroup
 
-  github_group "acir_tests building browser-test-app"
-  (cd browser-test-app && yarn build)
+  denoise "cd browser-test-app && yarn build"
+
   github_endgroup
 }
 
 function hash {
-  cache_content_hash ../../noir/.rebuild_patterns ../../noir/.rebuild_patterns_tests ../../barretenberg/cpp/.rebuild_patterns ../../barretenberg/ts/.rebuild_patterns
+  cache_content_hash \
+    ../../noir/.rebuild_patterns \
+    ../../noir/.rebuild_patterns_tests \
+    ../../barretenberg/cpp/.rebuild_patterns \
+    ../../barretenberg/ts/.rebuild_patterns
 }
+
 function test {
   set -eu
-  github_group "acir_tests testing"
+
   local hash=$(hash)
-  if ! test_should_run barretenberg-acir-tests-$hash; then
-    github_endgroup
-    return
-  fi
+  test_should_run barretenberg-acir-tests-$hash || return 0
+
+  github_group "acir_tests testing"
 
   # TODO: These are some magic numbers that fit our dev/ci environments. They ultimately need to work on lower hardware.
   export HARDWARE_CONCURRENCY=${HARDWARE_CONCURRENCY:-8}
   # local jobs=$(($(nproc) / HARDWARE_CONCURRENCY))
   local jobs=64
 
-  # Create temporary file descriptor 3, and redirects anything written to it, to parallels stdin.
-  exec 3> >(parallel -j$jobs --tag --line-buffered --joblog joblog.txt)
-  local pid=$!
-  trap "kill -SIGTERM $pid 2>/dev/null || true" EXIT
+  test_cmds | (cd $root; parallel -j$jobs --tag --line-buffered --joblog joblog.txt)
 
-  # Run function for syntactic simplicity.
-  run() {
-      echo "$*" >&3
-  }
+  cache_upload_flag barretenberg-acir-tests-$hash
+  github_endgroup
+}
 
+# Prints to stdout, one per line, the command to execute each individual test.
+# Paths are all relative to the repository root.
+function test_cmds {
   local plonk_tests=$(find ./acir_tests -maxdepth 1 -mindepth 1 -type d | \
     grep -vE 'verify_honk_proof|double_verify_honk_proof|verify_rollup_honk_proof')
   local honk_tests=$(find ./acir_tests -maxdepth 1 -mindepth 1 -type d | \
     grep -vE 'single_verify_proof|double_verify_proof|double_verify_nested_proof|verify_rollup_honk_proof')
 
+  local run_test=$(realpath --relative-to=$root ./run_test.sh)
+  local run_test_browser=$(realpath --relative-to=$root ./run_test_browser.sh)
+  local bbjs_bin="../ts/dest/node/main.js"
+
   # barretenberg-acir-tests-sol:
-  run FLOW=sol ./run_test.sh assert_statement
-  run FLOW=sol ./run_test.sh double_verify_proof
-  run FLOW=sol ./run_test.sh double_verify_nested_proof
-  run FLOW=sol_honk ./run_test.sh assert_statement
-  run FLOW=sol_honk ./run_test.sh 1_mul
-  run FLOW=sol_honk ./run_test.sh slices
-  run FLOW=sol_honk ./run_test.sh verify_honk_proof
+  echo FLOW=sol $run_test assert_statement
+  echo FLOW=sol $run_test double_verify_proof
+  echo FLOW=sol $run_test double_verify_nested_proof
+  echo FLOW=sol_honk $run_test assert_statement
+  echo FLOW=sol_honk $run_test 1_mul
+  echo FLOW=sol_honk $run_test slices
+  echo FLOW=sol_honk $run_test verify_honk_proof
 
   # barretenberg-acir-tests-bb.js:
   # Browser tests.
-  run BROWSER=chrome THREAD_MODEL=mt PORT=8080 ./run_test_browser.sh verify_honk_proof
-  run BROWSER=chrome THREAD_MODEL=st PORT=8081 ./run_test_browser.sh 1_mul
-  run BROWSER=webkit THREAD_MODEL=mt PORT=8082 ./run_test_browser.sh verify_honk_proof
-  run BROWSER=webkit THREAD_MODEL=st PORT=8083 ./run_test_browser.sh 1_mul
-  # Run ecdsa_secp256r1_3x through bb.js on node to check 256k support.
-  run BIN=../ts/dest/node/main.js FLOW=prove_then_verify ./run_test.sh ecdsa_secp256r1_3x
-  # Run the prove then verify flow for UltraHonk. This makes sure we have the same circuit for different witness inputs.
-  run BIN=../ts/dest/node/main.js SYS=ultra_honk FLOW=prove_then_verify ./run_test.sh 6_array
-  # Run a single arbitrary test not involving recursion through bb.js for MegaHonk
-  run BIN=../ts/dest/node/main.js SYS=mega_honk FLOW=prove_and_verify ./run_test.sh 6_array
-  # Run 1_mul through bb.js build, all_cmds flow, to test all cli args.
-  run BIN=../ts/dest/node/main.js FLOW=all_cmds ./run_test.sh 1_mul
+  echo BROWSER=chrome THREAD_MODEL=mt PORT=8080 $run_test_browser verify_honk_proof
+  echo BROWSER=chrome THREAD_MODEL=st PORT=8081 $run_test_browser 1_mul
+  echo BROWSER=webkit THREAD_MODEL=mt PORT=8082 $run_test_browser verify_honk_proof
+  echo BROWSER=webkit THREAD_MODEL=st PORT=8083 $run_test_browser 1_mul
+  # echo ecdsa_secp256r1_3x through bb.js on node to check 256k support.
+  echo BIN=$bbjs_bin FLOW=prove_then_verify $run_test ecdsa_secp256r1_3x
+  # echo the prove then verify flow for UltraHonk. This makes sure we have the same circuit for different witness inputs.
+  echo BIN=$bbjs_bin SYS=ultra_honk FLOW=prove_then_verify $run_test 6_array
+  # echo a single arbitrary test not involving recursion through bb.js for MegaHonk
+  echo BIN=$bbjs_bin SYS=mega_honk FLOW=prove_and_verify $run_test 6_array
+  # echo 1_mul through bb.js build, all_cmds flow, to test all cli args.
+  echo BIN=$bbjs_bin FLOW=all_cmds $run_test 1_mul
 
   # barretenberg-acir-tests-bb:
   # Fold and verify an ACIR program stack using ClientIvc, recursively verify as part of the Tube circuit and produce and verify a Honk proof
-  run FLOW=prove_then_verify_tube ./run_test.sh 6_array
-  # Run 1_mul through native bb build, all_cmds flow, to test all cli args.
-  run FLOW=all_cmds ./run_test.sh 1_mul
+  echo FLOW=prove_then_verify_tube $run_test 6_array
+  # echo 1_mul through native bb build, all_cmds flow, to test all cli args.
+  echo FLOW=all_cmds $run_test 1_mul
 
   # barretenberg-acir-tests-bb-ultra-plonk:
   # Exclude honk tests.
   for t in $plonk_tests; do
-    run FLOW=prove_then_verify ./run_test.sh $(basename $t)
+    echo FLOW=prove_then_verify $run_test $(basename $t)
   done
-  run FLOW=prove_then_verify RECURSIVE=true ./run_test.sh assert_statement
-  run FLOW=prove_then_verify RECURSIVE=true ./run_test.sh double_verify_proof
+  echo FLOW=prove_then_verify RECURSIVE=true $run_test assert_statement
+  echo FLOW=prove_then_verify RECURSIVE=true $run_test double_verify_proof
 
   # barretenberg-acir-tests-bb-ultra-honk:
   # Exclude plonk tests.
   for t in $honk_tests; do
-    run SYS=ultra_honk FLOW=prove_then_verify ./run_test.sh $(basename $t)
+    echo SYS=ultra_honk FLOW=prove_then_verify $run_test $(basename $t)
   done
-  run SYS=ultra_honk FLOW=prove_then_verify RECURSIVE=true ./run_test.sh assert_statement
-  run SYS=ultra_honk FLOW=prove_then_verify RECURSIVE=true ./run_test.sh double_verify_honk_proof
-  run SYS=ultra_honk FLOW=prove_and_verify_program ./run_test.sh merkle_insert
-  run SYS=ultra_rollup_honk FLOW=prove_and_verify ./run_test.sh verify_rollup_honk_proof
-
+  echo SYS=ultra_honk FLOW=prove_then_verify RECURSIVE=true $run_test assert_statement
+  echo SYS=ultra_honk FLOW=prove_then_verify RECURSIVE=true $run_test double_verify_honk_proof
+  echo SYS=ultra_honk FLOW=prove_and_verify_program $run_test merkle_insert
+  echo SYS=ultra_rollup_honk FLOW=prove_then_verify $run_test verify_rollup_honk_proof
 
   # barretenberg-acir-tests-bb-client-ivc:
-  run FLOW=prove_then_verify_client_ivc ./run_test.sh 6_array
-  run FLOW=prove_then_verify_client_ivc ./run_test.sh databus
-  run FLOW=prove_then_verify_client_ivc ./run_test.sh databus_two_calldata
-
-  # Close parallels input file descriptor and wait for completion.
-  exec 3>&-
-  wait $pid
-
-  cache_upload_flag barretenberg-acir-tests-$hash
-  github_endgroup
+  echo FLOW=prove_then_verify_client_ivc $run_test 6_array
+  echo FLOW=prove_then_verify_client_ivc $run_test databus
+  echo FLOW=prove_then_verify_client_ivc $run_test databus_two_calldata
 }
 
-export -f build test
+export -f build_tests test
 
 case "$cmd" in
   "clean")
     git clean -fdx
     (cd ../../noir/noir-repo/test_programs/execution_success && git clean -fdx)
     ;;
-  ""|"fast")
+  ""|"fast"|"full")
     ;;
-  "full")
-    denoise build
+  "build-tests")
+    build_tests
     ;;
   "ci")
-    denoise build
-    denoise test
+    build_tests
+    test
     ;;
   "hash")
     hash
     ;;
   "test")
-    denoise test
+    test
+    ;;
+  "test-cmds")
+    test_cmds
     ;;
   *)
     echo "Unknown command: $cmd"

diff --git a/barretenberg/acir_tests/browser-test-app/yarn.lock b/barretenberg/acir_tests/browser-test-app/yarn.lock
@@ -7,7 +7,7 @@ __metadata:
 
 "@aztec/bb.js@file:../../ts::locator=browser-test-app%40workspace%3A.":
   version: 0.67.1
-  resolution: "@aztec/bb.js@file:../../ts#../../ts::hash=29e47a&locator=browser-test-app%40workspace%3A."
+  resolution: "@aztec/bb.js@file:../../ts#../../ts::hash=cd38cd&locator=browser-test-app%40workspace%3A."
   dependencies:
     comlink: "npm:^4.4.1"
     commander: "npm:^12.1.0"
@@ -17,7 +17,7 @@ __metadata:
     tslib: "npm:^2.4.0"
   bin:
     bb.js: ./dest/node/main.js
-  checksum: 10c0/c01128ff74f29b6bbc5c46362792525ef5612c5fc8787341551bcf457ba9816a971e24a74292ab230c47b0b9efe8d7e0d1cabd44247e1b6e718727d0b6372400
+  checksum: 10c0/c6c1476f5f5d5cc1ea7022043e00870ee0743fd73a532c171586ab74bac53f3888c648bd4057de5a602e4a556cbb5d91454f57e0875ab002ccc87e7f83f12e43
   languageName: node
   linkType: hard
 

diff --git a/barretenberg/acir_tests/run_test_browser.sh b/barretenberg/acir_tests/run_test_browser.sh
@@ -6,6 +6,8 @@ cleanup() {
 }
 trap cleanup EXIT
 
+cd $(dirname $0)
+
 # Skipping firefox because this headless firefox is so slow.
 export BROWSER=${BROWSER:-chrome,webkit}
 

diff --git a/barretenberg/cpp/bootstrap.sh b/barretenberg/cpp/bootstrap.sh
@@ -3,22 +3,8 @@ source $(git rev-parse --show-toplevel)/ci3/source_bootstrap
 
 cmd=${1:-}
 
-# Determine system.
-if [[ "$OSTYPE" == "darwin"* ]]; then
-  os=macos
-elif [[ "$OSTYPE" == "linux-gnu" ]]; then
-  os=linux
-elif [[ "$OSTYPE" == "linux-musl" ]]; then
-  os=linux
-else
-  echo "Unknown OS: $OSTYPE"
-  exit 1
-fi
-
-# Pick native toolchain.
 preset=clang16-assert
 pic_preset="clang16-pic"
-
 hash=$(cache_content_hash .rebuild_patterns)
 
 function build_native {
@@ -68,59 +54,67 @@ function build {
   github_group "bb cpp build"
   export preset pic_preset hash
   export -f build_native build_wasm build_wasm_threads
-  parallel --line-buffered -v --tag --memfree 8g denoise {} ::: build_native build_wasm build_wasm_threads
+  parallel --line-buffered -v --tag denoise {} ::: build_native build_wasm build_wasm_threads
   github_endgroup
 }
 
-function test {
-  if test_should_run barretenberg-test-$hash; then
-    github_group "bb test"
-
-    echo "Check formatting..."
-    ./format.sh check
-
-    echo "Building tests..."
-    denoise cmake --preset $preset -Bbuild "&&" cmake --build build
-
-    # Download ignition transcripts.
-    # TODO: Use the flattened crs. These old transcripts are a pain.
-    echo "Downloading srs..."
-    denoise "cd ./srs_db && ./download_ignition.sh 3 && ./download_grumpkin.sh"
-    if [ ! -d ./srs_db/grumpkin ]; then
-      # The Grumpkin SRS is generated manually at the moment, only up to a large enough size for tests
-      # If tests require more points, the parameter can be increased here. Note: IPA requires
-      # dyadic_circuit_size + 1 points so in general this number will be a power of two plus 1
-      cd ./build && cmake --build . --parallel --target grumpkin_srs_gen && ./bin/grumpkin_srs_gen 32769
-    fi
+function build_tests {
+  github_group "bb build tests"
+  denoise ./format.sh check
+  denoise cmake --preset $preset -Bbuild "&&" cmake --build build
+  # Download ignition transcripts. Only needed for tests.
+  # The actual bb binary uses the flat crs downloaded in barratenberg/bootstrap.sh to ~/.bb-crs.
+  # TODO: Use the flattened crs. These old transcripts are a pain.
+  denoise "cd ./srs_db && ./download_ignition.sh 3 && ./download_grumpkin.sh"
+}
 
-    echo "Testing..."
-    (cd build && GTEST_COLOR=1 denoise ctest -j32 --output-on-failure)
-    cache_upload_flag barretenberg-test-$hash
-    github_endgroup
-  fi
+function test {
+  test_should_run barretenberg-test-$hash || return 0
+  github_group "bb test"
+  (cd build && GTEST_COLOR=1 denoise ctest -j32 --output-on-failure)
+  cache_upload_flag barretenberg-test-$hash
+  github_endgroup
 }
 
 case "$cmd" in
   "clean")
     git clean -fdx
     ;;
   ""|"fast")
+    # Build bb and wasms. Can be incremental.
     build
     ;;
   "full")
+    # Deletes all build dirs and build bb and wasms from scratch.
     rm -rf build*
     build
     ;;
+  "build-tests")
+    # Build the entire native repo, including all tests and benchmarks.
+    build_tests
+    ;;
   "test")
+    # Run the tests. Assumes they've been (re)built with a call to build_tests.
     test
     ;;
   "ci")
     build
+    build_tests
     test
     ;;
   "hash")
     echo $hash
     ;;
+  "test-cmds")
+    # Print every individual test command. Can be fed into gnu parallel.
+    cd build
+    for bin in ./bin/*_tests; do
+      bin_name=$(basename $bin)
+      $bin --gtest_list_tests | \
+        awk -vbin=$bin_name '/^[a-zA-Z]/ {suite=$1} /^[ ]/ {print "barretenberg/cpp/scripts/run_test.sh " bin " " suite$1}' | \
+        sed 's/\.$//' | grep -v 'DISABLED_'
+    done
+    ;;
   *)
     echo "Unknown command: $cmd"
     exit 1

diff --git a/barretenberg/cpp/scripts/run_test.sh b/barretenberg/cpp/scripts/run_test.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+# This runs an individual test.
+# It's the script used by ./bootstrap.sh test-cmds.
+# It means we can return a concise, easy to read, easy to run command for reproducing a test run.
+set -eu
+
+cd $(dirname $0)/../build
+
+export GTEST_COLOR=1
+export HARDWARE_CONCURRENCY=8
+# export IGNITION_CRS_PATH="./barretenberg/cpp/srs_db/ignition"
+# export GRUMPKIN_CRS_PATH="./barretenberg/cpp/srs_db/grumpkin"
+
+./bin/$1 --gtest_filter=$2