Skip to content

Commit

Permalink
bb.js is standalone prover that handles noir constraint/witness outpu…
Browse files Browse the repository at this point in the history
…t. (#471)

* Initial

* Separate thread build dir.

* Large vector slab allocator.

* format msgpack serialization and excldue msgpack-c from clang-format (#467)

* patch: temporarily remove broken solidity ci (#470)

* progress.

* Sumcheck improvements (#455)

* convert partially evaluated polynomials from vectors to Polynomials and rename

* rename fold method to partially_evaluate

* static constexpr barycentric arrays

* change purported evaluations to claimed evaluations

* specify relations in Flavor

* working. but cant compute pk independently :/

* Fixed.

* Better key construction separation.

* Clean

* tests pass

* Commander

* Fix.

* Fix

* Fix serve.

* Fix

* More commands.

* Fixed a bug in biggroup tests (#478)

* Add back custom libc++ build.

* Fix.

* write_vk

* DSL: Add KeccakVar opcode (#476)

* add initial KeccakVar code

* add result field

* add keccak_var_constraints to fields

* Multi-constraint Relations (#444)

Allow for correct and efficient batching over identities in the Sumcheck relation

* Clean up srs package. Add a global crs that the wasm can init once.

* Add back vk serialization for a3

* Clean

* Exclude env from lib.

* feat(dsl)!: DSL recursion changes w/ bb.js (#485)

* Added recursion constraint into dsl + tests

* fix dsl ecdsa constraint. Added ecdsa dsl tests

* changed dsl recursion to pass proof/key as witnesses

* recursive verification key no longer encapsulates a native key

recursive verification key no longer encapsulates a reference string

(neither were fundamentally needed!)

* added serialization test for acir_proofs

* Added serialization methods into dsl for recursive proof composition

Added passing serialization tests

Fixed format of RecursionConstraint to be compatible with existing ACIR formatting

* added verificaiton key hash into RecursionConstraint

Can be used by backend to force recursive verification key to be a specific value (it is represented via witnesses and therefore is not by-default constraint to represent a specific circuit)

* fixed compiler errors in stdlib_recursion_tests

* feat(dsl)!: Noir recursion updates (#379)

* merge master in zw/noir-recursion

* add dsl files to commit

* namespace stuff and debugging

* c bind functions

* constraint test and comment removal

* revert some changes to RecursionConstraint while debugging serialization issues

* dispaly error when trying to use create_circuit_with_witness

* is_recursive flag as part of new_proof and verify_proof, new RecursiveProver type in dsl, and serialization changes for recursion acir_proofs methods

* remove debug output from TestSerializationWithRecursion

* master merge conflicts

* EnvReferenceString undefined reference to  error workaround

* add missing recursion_constraint field from acir_format tests

* add recursion_constraints field back in acir_proofs.test inner circuit

* fix inner circuit in RecursionConstraint.TestRecursionConstraint

* Empty-Commit

* add verify_recursive_proof method for simulating recursive verification in the ACVM

* add ecc module to env package to fix linking of ennv_load_prover_crs and env_load_verifier_crs

* add back reference strinng to stdlib recursion key to pass sol verifier generation tests

* remove prints from recursive verifier test

* fix dirty free for when serializing vk to fields, was working on macbook so not caught earlier, but need more clarity on ubuntu

* fix ecdsa tests after master merge

* missing keccak constraints fields in acir format and proofs tests

* one more missing keccak constraint

* mismatched acir format structs for gcc build

* missing block constraints in acir tests

* feat(dsl)!: Arbitrary depth recursion (#433)

* merge conflicts and small updates to get acir_proofs test passing with arbitrary depth cahnges

* inline with mv/noir-recursion and working double recursion acir_proofs serialization test

* cleanup and working towards supporting full nested proofs, still some bugs in acir_proofs test

* full recursive composition test working in acir_proofs

* use two public inptus for inner circuit

* delete commented out unnecessary acir proofs method

* made dummy transcript points unique to prevent point collisions

* handle nested proofs when exporting dummy transcript, export recursive proof indices when init vkey, fix full recursive comp test in acir_proofs

* update bindings on verify_recursive_proof to accept num public inputs

* missing acir_format field in tests

* missing one more correct acir_format struct in recursion constraint test

* cleanup and additional comment for recursion_constraint

* fix up comment in recursion_constraint

* remove unnecesary comments when we were including proof outputs as public inputs in the recursion constraint

---------

Co-authored-by: zac-williamson <[email protected]>

* move export key into dsl package

* chore: remove unused export key in recursion format from main proof system classes

* pr review: moved from_witness comment and renamed from_field_pt_vector

* moved export transcript in recursion format to DSL package

* move order of acir functions

* remove ecc bb_module declaration in env package

* chore: remove usage of magic numbers when slicing g1::affine_element into barretenberg::fr

* introduce NUM_AGGREGATION_ELEMENTS constant and use it through recursion constraint

* nit ASSERT(result != -1) in get_public_input_index

* remove unused found var

* cast -1

* fix up verify_recursive_proof and add a test for it

* moved from tempalte for has_valid_witness_assignments to a flag

* chore: add comments to AcirProofs.TestVerifyRecursiveProofPass

* update prove and verify exports to handle is_recursive

* remove install state

* remove debug output and use total_circuit_size

* remove verify_recursive_proof as we might not need it for simulation

* switch how we do is_recursive

* remove acir recursive simulation method

* changed wasi stubs and removed cout from acir_init_proving_key

* removed cout outs getting runtime error for func definition

* working recursion format funcs, but strangely getting errors about trying to invert zero in the field for dummy transcript and key

* most recent debug output, hitting bad alloc in compute_proving_key_base

* delete old comment

* bb.js cmd changes

* Lint and default fixes.

* cleanup info and debug comments

* add comment to recursion serialization methods

* match package.json on cl/acir_cbinds

* don't remove EventEmitter from bb_wasm.ts

* delete empty exports file

* use process.exit

* log

* add comment for why CIRCUIT_SIZE is 2**18

* mising keccak var constraints in new acir format tests

* Cleanup some serialization.

* Cleanup for publishing. README.

---------

Co-authored-by: zac-williamson <[email protected]>
Co-authored-by: Charlie Lye <[email protected]>

---------

Co-authored-by: Maxim Vezenov <[email protected]>
Co-authored-by: Maddiaa <[email protected]>
Co-authored-by: ledwards2225 <[email protected]>
Co-authored-by: Innokentii Sennovskii <[email protected]>
Co-authored-by: kevaundray <[email protected]>
Co-authored-by: zac-williamson <[email protected]>
  • Loading branch information
7 people authored Jun 1, 2023
1 parent 3be0eb6 commit f9bfbe7
Show file tree
Hide file tree
Showing 191 changed files with 5,738 additions and 3,059 deletions.
23 changes: 0 additions & 23 deletions .github/workflows/solidity_verifier_test.yml

This file was deleted.

3 changes: 2 additions & 1 deletion .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,8 @@
"solidity.compileUsingRemoteVersion": "v0.6.10+commit.00c0fcaf",
"search.exclude": {
"**/.yarn": true,
"**/.pnp.*": true
"**/.pnp.*": true,
"**/msgpack-c/**": true
},
"typescript.tsdk": "ts/.yarn/sdks/typescript/lib",
"typescript.enablePromptUseWorkspaceTsdk": true,
Expand Down
1 change: 1 addition & 0 deletions cpp/CMakePresets.json
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@
"displayName": "Build for pthread enabled WASM",
"description": "Build for pthread enabled WASM",
"inherits": "wasm",
"binaryDir": "build-wasm-threads",
"environment": {
"WASI_SDK_PREFIX": "${sourceDir}/src/wasi-sdk-20.0+threads",
"CMAKE_BUILD_TYPE": "Release"
Expand Down
4 changes: 4 additions & 0 deletions cpp/bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -66,5 +66,9 @@ cmake --build --preset $PRESET ${@/#/--target }
./scripts/install-wasi-sdk.sh

# Build WASM.
cmake --preset wasm
cmake --build --preset wasm

# Build WASM with new threading.
cmake --preset wasm-threads
cmake --build --preset wasm-threads
51 changes: 51 additions & 0 deletions cpp/scripts/build_libacvm_backend.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
#!/bin/bash
set -eu

BB_TARGETS=(
libenv.a
libcommon.a
libcrypto_blake2s.a
libcrypto_pedersen_hash.a
libcrypto_pedersen_commitment.a
libcrypto_keccak.a
libcrypto_schnorr.a
libcrypto_generators.a
libnumeric.a
)

CMD="cmake --preset wasm && cmake --build --preset wasm"
for target in "${BB_TARGETS[@]}"; do CMD="$CMD --target $target"; done
eval $CMD

cd ./build-wasm/lib

LIBS=(
$PWD/libenv.a
$PWD/libcommon.a
$PWD/libcrypto_blake2s.a
$PWD/libcrypto_pedersen_hash.a
$PWD/libcrypto_pedersen_commitment.a
$PWD/libcrypto_keccak.a
$PWD/libcrypto_schnorr.a
$PWD/libcrypto_generators.a
$PWD/libnumeric.a
$PWD/../../src/wasi-sdk-20.0/share/wasi-sysroot/lib/wasm32-wasi/libc++.a
$PWD/../../src/wasi-sdk-20.0/share/wasi-sysroot/lib/wasm32-wasi/libc++abi.a
)

rm -rf scratch
mkdir -p scratch
cd scratch

for LIB_FILE_PATH in "${LIBS[@]}"; do
LIB=$(basename $LIB_FILE_PATH)
echo Extracting lib: $LIB
mkdir $LIB
cd $LIB
ar x $LIB_FILE_PATH
cd ..
done

rm -f ../libacvm_backend.a
#../../../src/wasi-sdk-12.0/bin/ar rcs ../libxyz.a libcrypto_blake2s.a/* libc++.a/* libc++abi.a/* libcrypto_pedersen_commitment.a/*
find . -type f -print0 | xargs -0 ../../../src/wasi-sdk-20.0/bin/ar rcs ../libacvm_backend.a
3 changes: 1 addition & 2 deletions cpp/scripts/install-wasi-sdk.sh
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,5 @@ else
# Replace wasi-sysroot in wasi-sdk-20.0+threads with our custom build.
# It contains libc++ and a patch by yamt to improve thread join stability.
# Can remove once future releases are more stable.
# UPDATE: Commenting out as we switched to "parallel_for_mutex_pool" as this we actually not very stable.
# curl -s -L https://wasi-sdk.s3.eu-west-2.amazonaws.com/yamt-wasi-sysroot-20.0.threads.tgz | tar zxfv -
curl -s -L https://wasi-sdk.s3.eu-west-2.amazonaws.com/yamt-wasi-sysroot-20.0.threads.tgz | tar zxfv -
fi
72 changes: 64 additions & 8 deletions cpp/src/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,12 @@ if(CMAKE_CXX_COMPILER_ID MATCHES "GNU")
endif()

# We enable -O1 level optimsations, even when compiling debug wasm, otherwise we get "local count too large" at runtime.
# We prioritise reducing size of final artefacts in release with -Oz.
if(WASM)
set(CMAKE_CXX_FLAGS_DEBUG "-O1 -g")
set(CMAKE_C_FLAGS_DEBUG "-O1 -g")
set(CMAKE_CXX_FLAGS_RELEASE "-Oz -DNDEBUG")
set(CMAKE_C_FLAGS_RELEASE "-Oz -DNDEBUG")
add_link_options(-Wl,--export-memory,--import-memory,--stack-first,-z,stack-size=1048576,--max-memory=4294967296)
endif()

Expand Down Expand Up @@ -70,6 +73,10 @@ endif()

include(GNUInstallDirs)

# For this library we include everything but the env and wasi modules, as it is the responsibility of the
# consumer of this library to define how and in what environment its artefact will run.
# libbarretenberg + libwasi = a wasi "reactor" that implements it's own env (e.g. logstr), e.g. barretenberg.wasm.
# libbarretenberg + env = a wasi "command" that expects a full wasi runtime (e.g. wasmtime), e.g. test binaries.
message(STATUS "Compiling all-in-one barretenberg archive")
add_library(
barretenberg
Expand Down Expand Up @@ -108,24 +115,73 @@ add_library(
$<TARGET_OBJECTS:transcript_objects>
)

if(WASM)
add_executable(
barretenberg.wasm
$<TARGET_OBJECTS:wasi_objects>
)

target_link_libraries(barretenberg.wasm barretenberg)
# Small library to provide necessary primitives for rust crate.
add_library(
acvm_backend
$<TARGET_OBJECTS:env_objects>
$<TARGET_OBJECTS:common_objects>
$<TARGET_OBJECTS:numeric_objects>
$<TARGET_OBJECTS:ecc_objects>
$<TARGET_OBJECTS:crypto_blake2s_objects>
$<TARGET_OBJECTS:crypto_keccak_objects>
$<TARGET_OBJECTS:crypto_generators_objects>
$<TARGET_OBJECTS:crypto_schnorr_objects>
$<TARGET_OBJECTS:crypto_pedersen_hash_objects>
$<TARGET_OBJECTS:crypto_pedersen_commitment_objects>
)

if(WASM)
# With binaryen installed, it seems its wasm backend optimiser gets invoked automatically.
# Due to either a bug in the optimiser, or non-standards compliant c++ in crypto/aes, tests start failing with
# -O3 level optimisations. We force down to -O2 for current workaround.
# TODO: Time has passed, check if this is still needed.
# UPDATE: Uninstall binaryen and any need downstream.
set(CMAKE_CXX_FLAGS_RELEASE "-O2")

# When building this wasm "executable", we include the wasi module but exclude the env module.
# That's because we expect this wasm to be run as a wasi "reactor" and for the host environment
# to implement the functions in env.
add_executable(
barretenberg.wasm
$<TARGET_OBJECTS:wasi_objects>
$<TARGET_OBJECTS:acir_format_objects>
$<TARGET_OBJECTS:acir_proofs_objects>
$<TARGET_OBJECTS:common_objects>
$<TARGET_OBJECTS:crypto_aes128_objects>
$<TARGET_OBJECTS:crypto_blake2s_objects>
$<TARGET_OBJECTS:crypto_blake3s_objects>
$<TARGET_OBJECTS:crypto_ecdsa_objects>
$<TARGET_OBJECTS:crypto_generators_objects>
$<TARGET_OBJECTS:crypto_keccak_objects>
$<TARGET_OBJECTS:crypto_pedersen_commitment_objects>
$<TARGET_OBJECTS:crypto_pedersen_hash_objects>
$<TARGET_OBJECTS:crypto_schnorr_objects>
$<TARGET_OBJECTS:crypto_sha256_objects>
$<TARGET_OBJECTS:ecc_objects>
$<TARGET_OBJECTS:examples_objects>
$<TARGET_OBJECTS:honk_objects>
$<TARGET_OBJECTS:numeric_objects>
$<TARGET_OBJECTS:plonk_objects>
$<TARGET_OBJECTS:polynomials_objects>
$<TARGET_OBJECTS:proof_system_objects>
$<TARGET_OBJECTS:srs_objects>
$<TARGET_OBJECTS:stdlib_aes128_objects>
$<TARGET_OBJECTS:stdlib_blake2s_objects>
$<TARGET_OBJECTS:stdlib_blake3s_objects>
$<TARGET_OBJECTS:stdlib_keccak_objects>
$<TARGET_OBJECTS:stdlib_merkle_tree_objects>
$<TARGET_OBJECTS:stdlib_pedersen_commitment_objects>
$<TARGET_OBJECTS:stdlib_pedersen_hash_objects>
$<TARGET_OBJECTS:stdlib_primitives_objects>
$<TARGET_OBJECTS:stdlib_schnorr_objects>
$<TARGET_OBJECTS:stdlib_sha256_objects>
$<TARGET_OBJECTS:transcript_objects>
)

target_link_options(
barretenberg.wasm
PRIVATE
-nostartfiles -Wl,--whole-archive,--no-entry,--export-dynamic,--allow-undefined
-nostartfiles -Wl,--no-entry,--export-dynamic,--allow-undefined
)

if(INSTALL_BARRETENBERG)
Expand Down
4 changes: 2 additions & 2 deletions cpp/src/barretenberg/benchmark/pippenger_bench/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
#include "barretenberg/common/assert.hpp"
#include <cstdlib>
#include "barretenberg/ecc/curves/bn254/scalar_multiplication/scalar_multiplication.hpp"
#include "barretenberg/srs/reference_string/file_reference_string.hpp"
#include "barretenberg/srs/factories/file_crs_factory.hpp"
#include "barretenberg/polynomials/polynomial_arithmetic.hpp"

// #include <valgrind/callgrind.h>
Expand Down Expand Up @@ -41,7 +41,7 @@ constexpr size_t NUM_POINTS = 1 << 16;
std::vector<fr> scalars;
static barretenberg::evaluation_domain small_domain;
static barretenberg::evaluation_domain large_domain;
auto reference_string = std::make_shared<proof_system::FileReferenceString>(NUM_POINTS, "../srs_db/ignition");
auto reference_string = std::make_shared<barretenberg::srs::factories::FileProverCrs>(NUM_POINTS, "../srs_db/ignition");

const auto init = []() {
small_domain = barretenberg::evaluation_domain(NUM_POINTS);
Expand Down
9 changes: 7 additions & 2 deletions cpp/src/barretenberg/common/c_bind.cpp
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
#ifndef NO_MULTITHREADING
#include "./c_bind.hpp"
#include "./mem.hpp"
#include "./timer.hpp"
#include "./serialize.hpp"
#include "./slab_allocator.hpp"
#include <algorithm>
#include <thread>

Expand Down Expand Up @@ -31,8 +33,6 @@ void thread_test_abort_entry_point(void*)
std::abort();
}

extern "C" {

WASM_EXPORT void test_threads(uint32_t const* thread_num, uint32_t const* iterations, uint32_t* out)
{
info("test starting...");
Expand Down Expand Up @@ -66,4 +66,9 @@ WASM_EXPORT void test_abort()
info("test_abort aborting");
std::abort();
}

WASM_EXPORT void common_init_slab_allocator(uint32_t const* circuit_size)
{
barretenberg::init_slab_allocator(ntohl(*circuit_size));
}
#endif
5 changes: 2 additions & 3 deletions cpp/src/barretenberg/common/c_bind.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,10 @@
#include "./wasm_export.hpp"
#include "./serialize.hpp"

extern "C" {

WASM_EXPORT void test_threads(uint32_t const* threads, uint32_t const* iterations, uint32_t* out);

WASM_EXPORT void test_thread_abort();

WASM_EXPORT void test_abort();
}

WASM_EXPORT void common_init_slab_allocator(uint32_t const* circuit_size);
Loading

0 comments on commit f9bfbe7

Please sign in to comment.