chore!: Replace computing hashes in circuits wasm, with computing them in ts via bb.js pedersen call.

[charlielye]

• This was originally going to use our TS pedersen lib to do all pedersen computations, but wasm is 10x faster, so we still want to use wasm.
• Introduce elliptic version of pedersen as it can be much faster than @noble/curves (WARNING: Jest totally messes with benchmarks and can't be trusted for timings with this stuff). Leaving noble version in for now in case want to revisit, but not exported via index. The elliptic version is also just there as reference (maybe useful in future).
• The pedersen api required being passed a wasm, and that wasm was obtained via async call. This creates a tonne of async function overhead and is generally a bit of a faff.
• This adds foundation/crypto/pedersen/pedersen.wasm.ts (and defaults to using it via index.ts) that mimics our pure TS pedersen api, but uses the bb.js wasm.
• Adds a hand rolled synchronous pedersen bind caller to bb.js (I should probably bring back making bindgen produce the sync api, but also figured might adopt Adams msgpack approach so not worth effort).
• foundation/crypto/pedersen leverages top-level await to load the wasm as a global, so we can remove a tonne of async await calls.
• We cleanup some of the c_binds in barretenberg.
• We still use circuits.wasm for a couple of more complex calls ive not tackled yet, e.g. hashVK.
• We now have dependency on bb.js, pulled in via portal yarn thingy. What does this mean for releases? (edit: This would probably have broken canary. This PR disabled canary. chore: Disable canary. #3244)
• Pulls the bbmalloc WASM_EXPORTS into own header so it doesn't mess with the bindgen util.
• Broke some circular import dependencies as I was hitting weird errors.
• All pedersen calls now made on foundation/crypto/pedersen, got rid of circuits/barretenberg/crypto/pedersen wrapper.
• Enable colorizing debug output in e2e tests via DEBUG_COLOR=1.
• Change some test() to it() (was faffing with other test runner when profiling pedersen).
• Timer uses performance.now rather than Date.getTime()
• Introduces madge as a tool to detect circular deps.

TODO in subsequent PR's.

• Move all the hashing functions in abis.ts to exist as hash functions on the various types.
• Implement hash functions for e.g. vkHash and a couple others that were to complicated for doing right now.
• Move the types out of circuits.js into types project.
• Ultimately remove need for circuits.js / circuits.wasm?

[socket-security]

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Packages	Version	New capabilities	Transitives	Size	Publisher
@types/elliptic	6.4.16	None	+1	25.5 kB	types
@types/bn.js	5.1.3	None	+0	13.8 kB	types
madge	6.1.0	filesystem, environment	+70	58.3 MB	kamiazya
typescript	3.9.10	None	+0	54.1 MB	typescript-bot

🚮 Removed packages: [email protected]

[AztecBot]

Benchmark results

No metrics with a significant change found.

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit e8945f80 and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,444	179,588	716,132
l1_rollup_calldata_gas	222,972	868,196	3,449,432
l1_rollup_execution_gas	842,059	3,595,304	22,204,801
l2_block_processing_time_in_ms	2,020	7,599 (-1%)	30,047 (+1%)
note_successful_decrypting_time_in_ms	300	873 (-3%)	3,219 (-2%)
note_trial_decrypting_time_in_ms	3.30 (-17%)	81.8 (-22%)	134 (+1%)
l2_block_building_time_in_ms	13,178	52,234 (-2%)	209,447 (-1%)
l2_block_rollup_simulation_time_in_ms	11,832	46,908 (-2%)	188,206 (-1%)
l2_block_public_tx_process_time_in_ms	1,302 (-1%)	5,190 (-1%)	20,743 (-2%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	21,730 (+1%)	42,345 (-1%)
note_history_successful_decrypting_time_in_ms	2,030 (-1%)	3,931 (-5%)
note_history_trial_decrypting_time_in_ms	122 (+2%)	177 (+24%)
node_database_size_in_bytes	1,628,864	1,100,279
pxe_database_size_in_bytes	29,748	59,307

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	770	61,697	18,905
private-kernel-ordering	123 (-1%)	24,297	8,153
base-rollup	1,766	656,311	814
root-rollup	77.4 (-2%)	4,072	1,097
private-kernel-inner	786	81,568	18,905
public-kernel-private-input	41.1 (-2%)	41,519	18,841
public-kernel-non-first-iteration	25.9 (-3%)	41,497	18,841
merge-rollup	0.940 (-3%)	2,592	873

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	8,787	27,547

[ludamad]

commented code can be removed

[charlielye]

Left it in as my next PR will actually make that commented code work. Just commented because discovered it's too much work for this PR.

[ludamad]

same as above

[ludamad]

mind tl;dr'ing why we're switching field types?

[charlielye]

This was an oopsie. Fixed. It was actually another api call that was meant to become Fq.
Typescript didnt raise error due to Fq and Fr being structurally the same, although this PR changes that as I optimised Fr slightly (made it Uint8Array under the hood rather than bigint as the conversion seriously impacts performance when doing lots of hashing).

[ludamad]

I'll keep this review light on the principle of engineers reviewing their own work first and foremost, so kept the review to 'am I comfortable with what I read here'. I've made minor comments, otherwise I saw a lot of propagation of async/await removal and read the comments above, LGTM.

One thing missing: The motivation, just for posterity, is to move away off of circuits dependency gradually, right?

[charlielye]

One thing missing: The motivation, just for posterity, is to move away off of circuits dependency gradually, right?

Yeah. The ultimate reason is to have much more of a pure TypeScript reference client rather than this extremely strong coupling with the C++ code/wasm. So bb.js can hopefully act as a lib for building proofs with a simple api, noir has their TS packages we can use, and then hopefully all of our yarn-project can just be pure TS.

(edit: I see the argument that circuits.js was a circuits encapsulating library, but with circuits now being written in Noir, it doesn't make sense to have a wasm for defining how we hash all these structs)

A side reason for this is just having the wasm plumbed into a hash function we use all over the place seems a bit off. It's nicer when the api is how it is in this PR, as we can easily switch how the hash is computed. Wether its a wasm, TS, or something else, is now encapsulated within the foundation/crypto/pedersen module.