feat: noir_wasm compilation of noir programs #3272

dan-aztec · 2023-11-08T05:54:23Z

Was starting #3273 and ran into an issue - the noir_wasm CLI is configured for compiling azec.nr smart contracts only, so needs update to support bin package type.

This reverts commit 452b858.

dan-aztec · 2023-11-08T06:08:03Z

barretenberg/cpp/src/barretenberg/relations/utils.hpp

@@ -86,8 +86,7 @@ template <typename Flavor> class RelationUtils {
    template <typename... T>
    static constexpr void add_tuples(std::tuple<T...>& tuple_1, const std::tuple<T...>& tuple_2)
    {
-        auto add_tuples_helper = [&]<std::size_t... I>(std::index_sequence<I...>)
-        {
+        auto add_tuples_helper = [&]<std::size_t... I>(std::index_sequence<I...>) {


must have an auto formatter on, can revert

yarn-project/noir-compiler/src/contract-interface-gen/abi.ts

dan-aztec · 2023-11-14T12:45:11Z

we may not actually use this for protocol circuits, due to wasm overhead of using noir_wasm vs nargo - TBD

AztecBot · 2023-11-16T12:29:00Z

Benchmark results

Metrics with a significant change:

note_trial_decrypting_time_in_ms (8): 5.39 (-71%)
note_trial_decrypting_time_in_ms (32): 20.4 (+51%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 5cfbb68b and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,444	179,588	716,132
l1_rollup_calldata_gas	222,960	867,848	3,447,956
l1_rollup_execution_gas	842,047	3,594,956	22,203,325
l2_block_processing_time_in_ms	2,034	7,649 (-2%)	30,403
note_successful_decrypting_time_in_ms	302 (-4%)	909 (+1%)	3,390 (+3%)
note_trial_decrypting_time_in_ms	⚠️ 5.39 (-71%)	⚠️ 20.4 (+51%)	142 (+1%)
l2_block_building_time_in_ms	20,711 (-2%)	82,755 (-1%)	329,749
l2_block_rollup_simulation_time_in_ms	11,939 (-4%)	47,966	190,057 (-1%)
l2_block_public_tx_process_time_in_ms	8,722 (-1%)	34,643 (-1%)	139,144

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	22,267	43,370
note_history_successful_decrypting_time_in_ms	2,094 (-1%)	4,057 (-2%)
note_history_trial_decrypting_time_in_ms	127	156
node_database_size_in_bytes	1,626,822	1,096,198 (-1%)
pxe_database_size_in_bytes	29,748	59,307

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	778 (+1%)	61,697	18,905
private-kernel-ordering	129 (+1%)	24,297	8,153
base-rollup	1,768	656,428	873
root-rollup	176 (+4%)	4,072	1,097
private-kernel-inner	803 (+1%)	81,568	18,905
public-kernel-private-input	578 (+1%)	41,519	18,905
public-kernel-non-first-iteration	413	41,561	18,905
merge-rollup	15.7	2,592	873

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	8,787	27,547

dan-aztec · 2023-11-16T12:51:57Z

yarn-project/noir-compiler/src/noir_artifact.ts

+  artifact: NoirCompilationResult,
+): artifact is NoirContractCompilationArtifacts {
+  return (artifact as NoirContractCompilationArtifacts).contract !== undefined;
+}


first time trying to do type matching in typescript, so this may not be best way

The best way is a discriminated union, but if the relationship doesn't have a tag field made for discriminated union this can be the best you can do

dan-aztec · 2023-11-21T19:28:42Z

whoa failing due to some dependency cycles on CI now...

alexghr · 2023-11-21T19:32:53Z

yarn-project/noir-compiler/src/cli/compileNoir.ts

+interface options {
+  // eslint-disable-next-line jsdoc/require-jsdoc
+  outdir: string;
+  // eslint-disable-next-line jsdoc/require-jsdoc
+  typescript: string | undefined;
+  // eslint-disable-next-line jsdoc/require-jsdoc
+  interface: string | undefined;
+  // eslint-disable-next-line jsdoc/require-jsdoc
+  compiler: string | undefined;
+}


I think the convention is to use pascal case for interface names. Potentially we could simplify the lint rules too

Suggested change

interface options {

// eslint-disable-next-line jsdoc/require-jsdoc

outdir: string;

// eslint-disable-next-line jsdoc/require-jsdoc

typescript: string | undefined;

// eslint-disable-next-line jsdoc/require-jsdoc

interface: string | undefined;

// eslint-disable-next-line jsdoc/require-jsdoc

compiler: string | undefined;

}

interface Options {

/* eslint-disable jsdoc/require-jsdoc */

outdir: string;

typescript: string | undefined;

interface: string | undefined;

compiler: string | undefined;

/* eslint-enable jsdoc/require-jsdoc */

}

alexghr · 2023-11-21T19:38:18Z

yarn-project/noir-compiler/src/compile/noir/noir-wasm-compiler.ts

    this.#debugLog(`Compiling contract at ${this.#package.getEntryPointPath()}`);
    await this.#dependencyManager.resolveDependencies();
    this.#debugLog(`Dependencies: ${this.#dependencyManager.getPackageNames().join(', ')}`);

    initializeResolver(this.#resolveFile);

    try {
+      const isContract: boolean = true; // this.#package.getType() === 'contract';


WDYT about removing the commented out code at the end? I like the explicit name for the parameter (here and in compileProgram), makes it clear what we're passing.

oops yeah will remove!

alexghr · 2023-11-21T19:40:22Z

yarn-project/noir-compiler/src/contract-interface-gen/programTypescript.ts

-  if (type.kind !== 'struct') {
-    return result;
-  }
+  if (type.kind !== 'struct') {return result;}


I think Prettier won't like this if statement on one line with braces.

yeah weird, i think this was from yarn formatting:fix... will take a look

alexghr · 2023-11-21T19:43:13Z

yarn-project/noir-compiler/src/cli/compileNoir.ts

+  const contract = _result as ContractArtifact;
+  if (contract.name) {


I think this (and the if statement after it) could use the helper method isNoirContractCompilationArtifacts from noir_artifacts.ts

ahh i just tried, the types are actually slightly different... got bogged down converting ContractArtifact | ProgramArtifact type into NoirContractCompilationArtifacts | NoirProgramCompilationArtifacts

alexghr · 2023-11-21T19:48:32Z

yarn-project/noir-compiler/src/contract-interface-gen/programTypescript.ts

-const circuits = [
-  'private_kernel_init',
-  'private_kernel_inner',
-  'private_kernel_ordering',
-  'public_kernel_private_previous',
-  'public_kernel_public_previous',
-  'rollup_base',
-  'rollup_merge',
-  'rollup_root',
-];
-
-const main = async () => {
-  for (const circuit of circuits) {
-    const rawData = await fs.readFile(`./src/target/${circuit}.json`, 'utf-8');
-    const abiObj: NoirCompiledCircuit = JSON.parse(rawData);
-    const generatedInterface = generateTsInterface(abiObj.abi);
-    await fs.writeFile(`./src/types/${circuit}_types.ts`, generatedInterface);
-  }
-};
-
-try {
-  await main();
-} catch (err: unknown) {
-  log(`Error generating types ${err}`);
-  process.exit(1);
-}


I might be missing something, looks like this script was moved from noir-protocol-circuits to noir-compiler. Does it not need to execute these steps anymore?

Oh is it because the circuits are compiled with noir-compiler which will generate TS as part of the pipeline

ah yes it was moved that way, but we have not moved to noir-wasm for compiling protocol circuits (will see how compile times change, may stick with nargo if CI slows down).

so the "loop" and call logic remains in noir-protocol-circuits, so that the types are generated regardless of if we use noir-wasm or nargo (and we can remove it, if we switch to noir-wasm)

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.16.0</summary> ## [0.16.0](aztec-packages-v0.15.1...aztec-packages-v0.16.0) (2023-11-27) ### ⚠ BREAKING CHANGES * deprecate circuits/cpp ([#3421](#3421)) * call stack validation optimisation. ([#3387](#3387)) ### Features * Base rollup in noir ([#3257](#3257)) ([4a1e9c3](4a1e9c3)) * Call stack validation optimisation. ([#3387](#3387)) ([d06d5db](d06d5db)) * Goblin proof construction ([#3332](#3332)) ([6a7ebb6](6a7ebb6)) * More logs relevant for debugging failures of 2 pixies test ([#3370](#3370)) ([683a0f3](683a0f3)) * Noir subrepo. ([#3369](#3369)) ([d94d88b](d94d88b)) * Noir_wasm compilation of noir programs ([#3272](#3272)) ([f9981d5](f9981d5)) * Rollback public state changes on failure ([#3393](#3393)) ([0e276fb](0e276fb)) ### Bug Fixes * **docs:** Doc explaining noir debug_log ([#3322](#3322)) ([eed023d](eed023d)) * Naming inconsistency in private kernel ([#3384](#3384)) ([4743486](4743486)) * Race condition in `PXE.getTxReceipt(...)` ([#3411](#3411)) ([9557a66](9557a66)) ### Miscellaneous * Deprecate circuits/cpp ([#3421](#3421)) ([4973cfb](4973cfb)) * Deterministically deduplicate `cached_partial_non_native_field_multiplication` across wasm32 and native compilations ([#3425](#3425)) ([5524933](5524933)) * **docs:** Common patterns and anti patterns in aztec.nr ([#3413](#3413)) ([65bd855](65bd855)) * Fix and reenable e2e quick start ([#3403](#3403)) ([112740e](112740e)), closes [#3356](#3356) * Fix intermittent failures for block-building e2e test ([#3404](#3404)) ([e76e2d4](e76e2d4)), closes [#3358](#3358) * Formatted `noir-contracts` and `aztec-nr` ([a73c4aa](a73c4aa)) * Initial clone of noir to subrepo ([#3409](#3409)) ([8f1cb83](8f1cb83)) * **noir-contracts:** Remove redundant return value of 1 ([#3415](#3415)) ([2001d47](2001d47)), closes [#2615](#2615) * Plumbs noir subrepo into yarn-project. ([#3420](#3420)) ([63173c4](63173c4)) * Remove pxe / node /p2p-bootstrap docker images ([#3396](#3396)) ([c236143](c236143)) * Skip artifacts for prettier ([#3399](#3399)) ([98d9e04](98d9e04)) * Update path to acir artifacts ([#3426](#3426)) ([f56f88d](f56f88d)) </details> <details><summary>barretenberg.js: 0.16.0</summary> ## [0.16.0](barretenberg.js-v0.15.1...barretenberg.js-v0.16.0) (2023-11-27) ### Miscellaneous * Plumbs noir subrepo into yarn-project. ([#3420](#3420)) ([63173c4](63173c4)) </details> <details><summary>barretenberg: 0.16.0</summary> ## [0.16.0](barretenberg-v0.15.1...barretenberg-v0.16.0) (2023-11-27) ### Features * Goblin proof construction ([#3332](#3332)) ([6a7ebb6](6a7ebb6)) * Noir subrepo. ([#3369](#3369)) ([d94d88b](d94d88b)) ### Miscellaneous * Deterministically deduplicate `cached_partial_non_native_field_multiplication` across wasm32 and native compilations ([#3425](#3425)) ([5524933](5524933)) * Plumbs noir subrepo into yarn-project. ([#3420](#3420)) ([63173c4](63173c4)) * Update path to acir artifacts ([#3426](#3426)) ([f56f88d](f56f88d)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.16.0</summary> ## [0.16.0](AztecProtocol/aztec-packages@aztec-packages-v0.15.1...aztec-packages-v0.16.0) (2023-11-27) ### ⚠ BREAKING CHANGES * deprecate circuits/cpp ([#3421](AztecProtocol/aztec-packages#3421)) * call stack validation optimisation. ([#3387](AztecProtocol/aztec-packages#3387)) ### Features * Base rollup in noir ([#3257](AztecProtocol/aztec-packages#3257)) ([4a1e9c3](AztecProtocol/aztec-packages@4a1e9c3)) * Call stack validation optimisation. ([#3387](AztecProtocol/aztec-packages#3387)) ([d06d5db](AztecProtocol/aztec-packages@d06d5db)) * Goblin proof construction ([#3332](AztecProtocol/aztec-packages#3332)) ([6a7ebb6](AztecProtocol/aztec-packages@6a7ebb6)) * More logs relevant for debugging failures of 2 pixies test ([#3370](AztecProtocol/aztec-packages#3370)) ([683a0f3](AztecProtocol/aztec-packages@683a0f3)) * Noir subrepo. ([#3369](AztecProtocol/aztec-packages#3369)) ([d94d88b](AztecProtocol/aztec-packages@d94d88b)) * Noir_wasm compilation of noir programs ([#3272](AztecProtocol/aztec-packages#3272)) ([f9981d5](AztecProtocol/aztec-packages@f9981d5)) * Rollback public state changes on failure ([#3393](AztecProtocol/aztec-packages#3393)) ([0e276fb](AztecProtocol/aztec-packages@0e276fb)) ### Bug Fixes * **docs:** Doc explaining noir debug_log ([#3322](AztecProtocol/aztec-packages#3322)) ([eed023d](AztecProtocol/aztec-packages@eed023d)) * Naming inconsistency in private kernel ([#3384](AztecProtocol/aztec-packages#3384)) ([4743486](AztecProtocol/aztec-packages@4743486)) * Race condition in `PXE.getTxReceipt(...)` ([#3411](AztecProtocol/aztec-packages#3411)) ([9557a66](AztecProtocol/aztec-packages@9557a66)) ### Miscellaneous * Deprecate circuits/cpp ([#3421](AztecProtocol/aztec-packages#3421)) ([4973cfb](AztecProtocol/aztec-packages@4973cfb)) * Deterministically deduplicate `cached_partial_non_native_field_multiplication` across wasm32 and native compilations ([#3425](AztecProtocol/aztec-packages#3425)) ([5524933](AztecProtocol/aztec-packages@5524933)) * **docs:** Common patterns and anti patterns in aztec.nr ([#3413](AztecProtocol/aztec-packages#3413)) ([65bd855](AztecProtocol/aztec-packages@65bd855)) * Fix and reenable e2e quick start ([#3403](AztecProtocol/aztec-packages#3403)) ([112740e](AztecProtocol/aztec-packages@112740e)), closes [#3356](AztecProtocol/aztec-packages#3356) * Fix intermittent failures for block-building e2e test ([#3404](AztecProtocol/aztec-packages#3404)) ([e76e2d4](AztecProtocol/aztec-packages@e76e2d4)), closes [#3358](AztecProtocol/aztec-packages#3358) * Formatted `noir-contracts` and `aztec-nr` ([a73c4aa](AztecProtocol/aztec-packages@a73c4aa)) * Initial clone of noir to subrepo ([#3409](AztecProtocol/aztec-packages#3409)) ([8f1cb83](AztecProtocol/aztec-packages@8f1cb83)) * **noir-contracts:** Remove redundant return value of 1 ([#3415](AztecProtocol/aztec-packages#3415)) ([2001d47](AztecProtocol/aztec-packages@2001d47)), closes [#2615](AztecProtocol/aztec-packages#2615) * Plumbs noir subrepo into yarn-project. ([#3420](AztecProtocol/aztec-packages#3420)) ([63173c4](AztecProtocol/aztec-packages@63173c4)) * Remove pxe / node /p2p-bootstrap docker images ([#3396](AztecProtocol/aztec-packages#3396)) ([c236143](AztecProtocol/aztec-packages@c236143)) * Skip artifacts for prettier ([#3399](AztecProtocol/aztec-packages#3399)) ([98d9e04](AztecProtocol/aztec-packages@98d9e04)) * Update path to acir artifacts ([#3426](AztecProtocol/aztec-packages#3426)) ([f56f88d](AztecProtocol/aztec-packages@f56f88d)) </details> <details><summary>barretenberg.js: 0.16.0</summary> ## [0.16.0](AztecProtocol/aztec-packages@barretenberg.js-v0.15.1...barretenberg.js-v0.16.0) (2023-11-27) ### Miscellaneous * Plumbs noir subrepo into yarn-project. ([#3420](AztecProtocol/aztec-packages#3420)) ([63173c4](AztecProtocol/aztec-packages@63173c4)) </details> <details><summary>barretenberg: 0.16.0</summary> ## [0.16.0](AztecProtocol/aztec-packages@barretenberg-v0.15.1...barretenberg-v0.16.0) (2023-11-27) ### Features * Goblin proof construction ([#3332](AztecProtocol/aztec-packages#3332)) ([6a7ebb6](AztecProtocol/aztec-packages@6a7ebb6)) * Noir subrepo. ([#3369](AztecProtocol/aztec-packages#3369)) ([d94d88b](AztecProtocol/aztec-packages@d94d88b)) ### Miscellaneous * Deterministically deduplicate `cached_partial_non_native_field_multiplication` across wasm32 and native compilations ([#3425](AztecProtocol/aztec-packages#3425)) ([5524933](AztecProtocol/aztec-packages@5524933)) * Plumbs noir subrepo into yarn-project. ([#3420](AztecProtocol/aztec-packages#3420)) ([63173c4](AztecProtocol/aztec-packages@63173c4)) * Update path to acir artifacts ([#3426](AztecProtocol/aztec-packages#3426)) ([f56f88d](AztecProtocol/aztec-packages@f56f88d)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

dan-aztec added 23 commits November 6, 2023 14:09

stop committing artifacts

41413c5

undo box artifact changes

b2c5741

compile non-contract package types with noir-wasm-compiler.ts

9abd0ff

revert "binary" to "bin"

b51c19a

bin

e44955d

revert to nargo for compiler

fa43b73

use types, still broken

27c62b2

remove src/target from git

ae7ef52

handle non-contract return type

94d6399

generate types

1473663

handle program differently from contract

3a9bd7e

handle program and contract separately

d62887e

update test

cde221e

stop logging

d26d6ac

fix type

d1b1d4e

switch command name to compile

c34bb69

readd src/target for testing

c79086e

point outdir to src/target

d1f1ecb

single build, and absoltue path to /src/target

e655751

merge conflicts

7ebbf5d

checkout origin/master barretenberg?

452b858

fetch

c635a05

Revert "checkout origin/master barretenberg?"

e2c70da

This reverts commit 452b858.

dan-aztec commented Nov 8, 2023

View reviewed changes

dan-aztec added 2 commits November 8, 2023 09:09

revert noir-protocol-circuits changes

eae79ec

origin not local

cf82db1

dan-aztec commented Nov 8, 2023

View reviewed changes

yarn-project/noir-compiler/src/contract-interface-gen/abi.ts Outdated Show resolved Hide resolved

revert more protocol circuits changes

1f38e39

dan-aztec requested a review from alexghr November 8, 2023 06:17

dan-aztec changed the title ~~feat: noira_wasm compilation of noir programs~~ feat: noir_wasm compilation of noir programs Nov 8, 2023

dan-aztec and others added 6 commits November 14, 2023 16:08

rename to compileNoir

71ba52f

Merge branch 'master' into dan/build-artifacts-ci-merge-conflicgt

0911592

update yarn command to compile

955417e

revert formatting

8efa75a

Merge branch 'master' into dan/build-artifacts-ci-merge-conflicgt

47b8c2b

update compileContract name

af5832f

dan-aztec commented Nov 16, 2023

View reviewed changes

dan-aztec and others added 5 commits November 17, 2023 00:50

Merge branch 'master' into dan/build-artifacts-ci-merge-conflicgt

eb6f46e

merge conflict

d7abdb5

small change to output json data

c2659eb

move generateTypescript logic out of noir-protocol-circuits

857101e

write program types to ../types/

d3a51cc

dan-aztec marked this pull request as ready for review November 18, 2023 09:38

dan-aztec added 2 commits November 21, 2023 10:42

remove formatted deleted code

2dde02d

revert cpp format

d2af58d

revert package.json change

d5ada69

alexghr approved these changes Nov 21, 2023

View reviewed changes

dan-aztec added 2 commits November 21, 2023 12:03

formatting

ac66988

formatting

b0be295

dan-aztec enabled auto-merge (squash) November 21, 2023 22:23

dan-aztec disabled auto-merge November 22, 2023 06:38

dan-aztec merged commit f9981d5 into master Nov 22, 2023
77 of 78 checks passed

dan-aztec deleted the dan/build-artifacts-ci-merge-conflicgt branch November 22, 2023 06:40

AztecBot mentioned this pull request Nov 22, 2023

chore(master): Release 0.16.0 #3385

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: noir_wasm compilation of noir programs #3272

feat: noir_wasm compilation of noir programs #3272

dan-aztec commented Nov 8, 2023 •

edited

Loading

dan-aztec Nov 8, 2023

dan-aztec commented Nov 14, 2023

AztecBot commented Nov 16, 2023 •

edited

Loading

L2 block published to L1

L2 chain processing

Circuits stats

Miscellaneous

dan-aztec Nov 16, 2023

ludamad Nov 16, 2023

dan-aztec commented Nov 21, 2023

alexghr Nov 21, 2023

alexghr Nov 21, 2023

dan-aztec Nov 21, 2023

alexghr Nov 21, 2023

dan-aztec Nov 21, 2023

alexghr Nov 21, 2023

dan-aztec Nov 21, 2023

alexghr Nov 21, 2023

alexghr Nov 21, 2023

dan-aztec Nov 21, 2023

		const contract = _result as ContractArtifact;
		if (contract.name) {

feat: noir_wasm compilation of noir programs #3272

feat: noir_wasm compilation of noir programs #3272

Conversation

dan-aztec commented Nov 8, 2023 • edited Loading

Choose a reason for hiding this comment

dan-aztec commented Nov 14, 2023

AztecBot commented Nov 16, 2023 • edited Loading

Benchmark results

L2 block published to L1

L2 chain processing

Circuits stats

Miscellaneous

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dan-aztec commented Nov 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dan-aztec commented Nov 8, 2023 •

edited

Loading

AztecBot commented Nov 16, 2023 •

edited

Loading