chore!: Remove recursive from ACIR format; add them to API and CLI …

…(#9479)

Resolves noir-lang/noir#6185

* Remove the `recursive` field from ACIR formats and the `Circuit`
definition
* Add `--recursive` to `bb` CLI
* Add `recursive` to `main.ts`, the backend API and Wasm

This is effectively undoing a lot of what was done
[here](AztecProtocol/aztec-packages@9c965a7#diff-2b9fe3a6f248b96aefc37782cc4c321567eed5dd10ab24472620a68c0fb4506bR29).
Interestingly there many more Wasm methods that need the `recursive`
parameter now: whereas previously only `acir_create_proof` and
`acir_verify_proof` used it, now anything that calls `create_circuit`
needs it because circuit creation builds all the constraints already,
which depend on this.

TODO:
- [x] Remove the `#[recursive]` attribute from Noir
- [x] Should the "prove and verify" methods that return nothing have a
recursive parameter?
- [x] Remove `#[recursive]` from `noir-protocol-circuits`
- [x] Update `bb-prover` where it uses uses `noir-protocol-circuits` to
use the correct `recursive` flag
- [x] Add `--recursive` to `cli.ts` under `bb-prover`
- [x] Check all calls to `executeBB` and pass `--recursive` if it calls
a `bb` command that needs it

---------

Co-authored-by: Tom French <[email protected]>
Co-authored-by: Tom French <[email protected]>

Earthfile

@@ -50,6 +50,7 @@ barretenberg-acir-tests-bb-ultra-plonk:
     # Run every acir test through native bb build prove_then_verify flow for UltraPlonk.
     # This ensures we test independent pk construction through real/garbage witness data paths.
     RUN FLOW=prove_then_verify ./run_acir_tests.sh
+    RUN FLOW=prove_then_verify RECURSIVE=true ./run_acir_tests.sh assert_statement double_verify_proof
 
 barretenberg-acir-tests-bb-ultra-honk:
     FROM ../build-images/+from-registry
@@ -67,6 +68,7 @@ barretenberg-acir-tests-bb-ultra-honk:
     # Run the acir test through native bb build prove_then_verify_ultra_honk flow
     # Note that the script will skip the Plonk related tests
     RUN FLOW=prove_then_verify_ultra_honk HONK=true ./run_acir_tests.sh
+    RUN FLOW=prove_then_verify_ultra_honk HONK=true RECURSIVE=true ./run_acir_tests.sh assert_statement double_verify_honk_proof
 
     # Construct and verify a UltraHonk proof for a single program
     RUN FLOW=prove_and_verify_ultra_honk ./run_acir_tests.sh pedersen_hash
@@ -173,4 +175,4 @@ barretenberg-acir-tests-bb.js:
     # Run fold_basic test through bb.js which runs ClientIVC on fold basic
     RUN BIN=../ts/dest/node/main.js FLOW=fold_and_verify_program ./run_acir_tests.sh fold_basic
     # Run 1_mul through bb.js build, all_cmds flow, to test all cli args.
-    RUN BIN=../ts/dest/node/main.js FLOW=all_cmds ./run_acir_tests.sh 1_mul
+    RUN BIN=../ts/dest/node/main.js FLOW=all_cmds ./run_acir_tests.sh 1_mul

acir_tests/Dockerfile.bb.sol

@@ -18,8 +18,7 @@ COPY . .
 # This includes the basic `assert_statement` test that contains a single public input
 # and the recursive aggregation circuits which use the Keccak based prover.
 #
-# NOTE: When circuits are marked `recursive` it means the backend will use a prover that
-# produces SNARK recursion friendly proofs, while the solidity verifier expects proofs
-# whose transcript uses Keccak hashing. 
+# NOTE: The solidity verifier expects proofs whose transcript uses Keccak hashing,
+# for which we have to invoke the backend prover without the `--recursive` flag.
 RUN (cd sol-test && yarn)
 RUN PARALLEL=1 FLOW=sol ./run_acir_tests.sh assert_statement double_verify_proof double_verify_nested_proof

acir_tests/README.md

@@ -49,14 +49,14 @@ $ FLOW=all_cmds ./run_acir_tests.sh 1_mul
 
 This means we have to generate the proof specific inputs using our backend and pass it back into `double_verify_proof` to regenerate the accurate witness. The following is a temporary solution to manually regenerate the inputs for `double_verify_proof` on a specific Noir branch.
 
-First find `acir_tests/gen_inner_proof_inputs.sh`. Change the $BRANCH env var to your working branch and $PROOF_NAME to your first input you want to recursively verify. The script is going to generate the proof system specific verification key output and proof for the `assert_statement_recursive` test. 
+First find `acir_tests/gen_inner_proof_inputs.sh`. Change the $BRANCH env var to your working branch and $PROOF_NAME to your first input you want to recursively verify. The script is going to generate the proof system specific verification key output and proof for the `assert_statement` test.
 
 To run:
 ```
 ./gen_inner_proof_inputs.sh
 ```
-To generate a new input you can run the script again. To generate a new file under `assert_statement_recursive/proofs/` be sure to change the $PROOF_NAME inside of the script.
+To generate a new input you can run the script again. To generate a new file under `assert_statement/proofs/` be sure to change the $PROOF_NAME inside of the script.
 
 You can then copy these inputs over to your working branch in Noir and regenerate the witness for `double_verify_proof`. You can then change the branch in `run_acir_tests.sh` to this Noir working branch as well and `double_verify_proof` should pass.
 
-The same process should then be repeated, but now `double_verify_proof_recursive` will be the circuit for which we will be generating recursive inputs using `gen_inner_proof_inputs.sh`. The recursive artifacts should then supplied as inputs to `double_verify_nested_proof`. 
+The same process should then be repeated, but now `double_verify_proof_recursive` will be the circuit for which we will be generating recursive inputs using `gen_inner_proof_inputs.sh`. The recursive artifacts should then supplied as inputs to `double_verify_nested_proof`.

acir_tests/flows/prove_and_verify.sh

@@ -2,7 +2,11 @@
 set -eu
 
 VFLAG=${VERBOSE:+-v}
+FLAGS="-c $CRS_PATH $VFLAG"
+if [ "${RECURSIVE}" = "true" ]; then
+    FLAGS="$FLAGS --recursive"
+fi
 
 # This is the fastest flow, because it only generates pk/vk once, gate count once, etc.
 # It may not catch all class of bugs.
-$BIN prove_and_verify $VFLAG -c $CRS_PATH -b ./target/program.json
+$BIN prove_and_verify $FLAGS -b ./target/program.json

acir_tests/flows/prove_and_verify_ultra_honk.sh

@@ -2,5 +2,9 @@
 set -eu
 
 VFLAG=${VERBOSE:+-v}
+FLAGS="-c $CRS_PATH $VFLAG"
+if [ "${RECURSIVE}" = "true" ]; then
+    FLAGS="$FLAGS --recursive"
+fi
 
-$BIN prove_and_verify_ultra_honk $VFLAG -c $CRS_PATH -b ./target/program.json
+$BIN prove_and_verify_ultra_honk $FLAGS -b ./target/program.json

acir_tests/flows/prove_and_verify_ultra_honk_program.sh

@@ -2,5 +2,9 @@
 set -eu
 
 VFLAG=${VERBOSE:+-v}
+FLAGS="-c $CRS_PATH $VFLAG"
+if [ "${RECURSIVE}" = "true" ]; then
+    FLAGS="$FLAGS --recursive"
+fi
 
-$BIN prove_and_verify_ultra_honk_program $VFLAG -c $CRS_PATH -b ./target/program.json
+$BIN prove_and_verify_ultra_honk_program $FLAGS -b ./target/program.json

acir_tests/flows/prove_then_verify.sh

@@ -4,6 +4,9 @@ set -eu
 VFLAG=${VERBOSE:+-v}
 BFLAG="-b ./target/program.json"
 FLAGS="-c $CRS_PATH $VFLAG"
+if [ "${RECURSIVE}" = "true" ]; then
+    FLAGS="$FLAGS --recursive"
+fi
 
 # Test we can perform the proof/verify flow.
 # This ensures we test independent pk construction through real/garbage witness data paths.

acir_tests/flows/prove_then_verify_ultra_honk.sh

@@ -4,6 +4,9 @@ set -eux
 VFLAG=${VERBOSE:+-v}
 BFLAG="-b ./target/program.json"
 FLAGS="-c $CRS_PATH $VFLAG"
+if [ "${RECURSIVE}" = "true" ]; then
+    FLAGS="$FLAGS --recursive"
+fi
 
 # Test we can perform the proof/verify flow.
 # This ensures we test independent pk construction through real/garbage witness data paths.

acir_tests/gen_inner_proof_inputs.sh

@@ -20,23 +20,23 @@ export BRANCH
 
 ./clone_test_vectors.sh
 
-cd acir_tests/assert_statement_recursive
+cd acir_tests/assert_statement
 
 PROOF_DIR=$PWD/proofs
 PROOF_PATH=$PROOF_DIR/$PROOF_NAME
 VFLAG=${VERBOSE:+-v}
 RFLAG=${RECURSIVE:+-r}
 
 echo "Write VK to file for assert_statement..."
-$BIN write_vk $VFLAG -c $CRS_PATH -o ./target/vk
+$BIN write_vk $VFLAG -c $CRS_PATH -o ./target/vk --recursive
 
 echo "Write VK as fields for recursion..."
 $BIN vk_as_fields $VFLAG -c $CRS_PATH -k ./target/vk -o ./target/vk_fields.json
 
 echo "Generate proof to file..."
 [ -d "$PROOF_DIR" ] || mkdir $PWD/proofs
 [ -e "$PROOF_PATH" ] || touch $PROOF_PATH
-$BIN prove $VFLAG -c $CRS_PATH -b ./target/program.json -o "./proofs/$PROOF_NAME"
+$BIN prove $VFLAG -c $CRS_PATH -b ./target/program.json -o "./proofs/$PROOF_NAME" --recursive
 
 echo "Write proof as fields for recursion..."
 $BIN proof_as_fields $VFLAG -c $CRS_PATH -p "./proofs/$PROOF_NAME" -k ./target/vk -o "./proofs/${PROOF_NAME}_fields.json"

acir_tests/regenerate_verify_honk_proof_inputs.sh

@@ -17,7 +17,7 @@ fi
 export BRANCH
 
 # the program for which a proof will be recursively verified
-PROGRAM=assert_statement_recursive
+PROGRAM=assert_statement
 # the program containing the recursive verifier
 RECURSIVE_PROGRAM=verify_honk_proof
 

acir_tests/run_acir_tests.sh

@@ -2,6 +2,7 @@
 # Env var overrides:
 #   BIN: to specify a different binary to test with (e.g. bb.js or bb.js-dev).
 #   VERBOSE: to enable logging for each test.
+#   RECURSIVE: to enable --recursive for each test.
 set -eu
 
 # Catch when running in parallel
@@ -19,6 +20,7 @@ VERBOSE=${VERBOSE:-}
 TEST_NAMES=("$@")
 # We get little performance benefit over 16 cores (in fact it can be worse).
 HARDWARE_CONCURRENCY=${HARDWARE_CONCURRENCY:-16}
+RECURSIVE=${RECURSIVE:-false}
 
 FLOW_SCRIPT=$(realpath ./flows/${FLOW}.sh)
 
@@ -28,7 +30,7 @@ else
     BIN=$(realpath $(which $BIN))
 fi
 
-export BIN CRS_PATH VERBOSE BRANCH
+export BIN CRS_PATH VERBOSE BRANCH RECURSIVE
 
 # copy the gzipped acir test data from noir/noir-repo/test_programs to barretenberg/acir_tests
 ./clone_test_vectors.sh
@@ -47,12 +49,12 @@ SKIP_ARRAY+=(regression_5045)
 # if HONK is false, we should skip verify_honk_proof
 if [ "$HONK" = false ]; then
     # Don't run programs with Honk recursive verifier
-    SKIP_ARRAY+=(verify_honk_proof double_verify_honk_proof double_verify_honk_proof_recursive)
+    SKIP_ARRAY+=(verify_honk_proof double_verify_honk_proof)
 fi
 
 if [ "$HONK" = true ]; then
     # Don't run programs with Plonk recursive verifier(s)
-    SKIP_ARRAY+=(single_verify_proof double_verify_proof double_verify_proof_recursive double_verify_nested_proof)
+    SKIP_ARRAY+=(single_verify_proof double_verify_proof double_verify_nested_proof)
 fi
 
 function test() {