feat: Noir subrepo. #3369

charlielye · 2023-11-20T20:47:08Z

Finally.

Builds the now subrepo'd noir repository. At present this just builds nargo for x86 and arm, meaning our sandbox install script can now provide a wrapper aztec-nargo that is guaranteed to always be exactly what we want, regardless of any modifications to macros, compilers, packages, or whatever. Decouples aztec from being dependent on Noirs deployment pipeline, while still providing the ability to push changes back to Noir.

Still need to do work to build npm packages and portal them into yarn-project. Will do in separate PR.

Removes old benchmarking stuff and commented canary stuff from pipeline.
Adds nargo x86 and arm builds, outputing a multiarch image that also runs efficiently on macs.
Removes a load of ARCH specific hack stuff in build-system, in favour of all cache image URI's now just have their arch appended to their tag. The arch is determined by the arch of the build system. This required a small "hack(?)" whereby we will fall back from an unfound arch (arm) to x86, as we build some of our arm images from previously build x86 images, but i anticipate that'll change at some point.
Remove some project scripts that bleed the build-system abstraction (e.g. deploy_docker.sh).
Introduces [ci dry-deploy] commit message command for doing dry run of deploys. We no longer have conditional workflow filters for deployment to enable this. I.e. there is always deploy jobs, they just noop out asap. If dry-deploy is enabled, the deploy jobs will run to "completion" but doesn't actually execute pushes to dockerhub, and just runs npm publish in dry-run mode.
Gets rid of VERSION files and the sanity check is it got in the way of the above, and I don't even remember why it was there.
build-system can now launch arm spot instances. This can be requested e.g. cond_spot_run_build noir 32 arm64.
Added a script to help bootstrap build-instances in case we need to create new AMI's again in future (I had to create and arm one).
Deleted some build-system scripts I couldn't see used anywhere.
deploy_dockerhub script now takes list of arches and does the manifest generation itself.
Introduces a should_deploy script which enables the early out of deployment jobs. At present this causes exit if there is no COMMIT_TAG. In future it'll want to run the deploy steps if BRANCH is master as well.

Follow up PRs will:

Build NPM packages and portal them into yarn-project.
Modify sandbox install script to pull noir image, install aztec-nargo wrapper.
Update vscode plugin to allow selection between global nargo or aztec-nargo if both found.
Someone will do some magic to use the nargo container in github codespaces so users can press . and play.

subrepo: subdir: "noir" merged: "d27db332f" upstream: origin: "[email protected]:noir-lang/noir" branch: "master" commit: "d27db332f" git-subrepo: version: "0.4.6" origin: "https://github.com/ingydotnet/git-subrepo" commit: "110b9eb"

ludamad · 2023-11-20T22:43:33Z

subrepo: subdir: "noir" merged: "816deeb60" upstream: origin: "[email protected]:noir-lang/noir" branch: "master" commit: "63d414c06" git-subrepo: version: "0.4.6" origin: "https://github.com/ingydotnet/git-subrepo" commit: "110b9eb"

subrepo: subdir: "noir" merged: "00af10514" upstream: origin: "[email protected]:noir-lang/noir" branch: "master" commit: "251b930de" git-subrepo: version: "0.4.6" origin: "https://github.com/ingydotnet/git-subrepo" commit: "110b9eb"

… out, unless dry deploy is set.

spypsy · 2023-11-23T13:19:14Z

.circleci/config.yml

+          command: |
+            [ -n "$COMMIT_TAG" ] || exit 0
+            deploy_dockerhub noir x86_64,arm64
+            deploy_dockerhub pxe x86_64,arm64


pxe image has been removed as it can now be started from the aztec-sandbox image with MODE=pxe

👍
My dry-deploy should test this and fail accordingly. At present it failed before even getting to the pxe line.
This dry-deploy stuff should be helpful...

spypsy · 2023-11-23T14:22:45Z

build_manifest.yml

+
+noir:
+  buildDir: noir
+  arch:


Is this used somewhere?

Actually no. Well spotted. Intention is to replace some of the hardcoded stuff in config.yml with a manifest query. e.g.

create_ecr_manifest aztec-sandbox x86_64,arm64 create_ecr_manifest cli x86_64,arm64

becomes

create_ecr_manifest aztec-sandbox create_ecr_manifest cli

And ultimately I'm still hopeful of fully code generating config.yml from the build manifest.

Maddiaa0 · 2023-11-23T14:56:37Z

build-system/scripts/cond_spot_run_script

-BASE_TAG=cache-$CONTENT_HASH
+# If the CPUS have a specific architecture assigned, we need to use that to build the success tag.
+ARR=(${CPUS//:/ })
+ARCH=${ARR[1]:-x86_64}


do we want to keep this long term, will not block this pr

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.16.0</summary> ## [0.16.0](aztec-packages-v0.15.1...aztec-packages-v0.16.0) (2023-11-27) ### ⚠ BREAKING CHANGES * deprecate circuits/cpp ([#3421](#3421)) * call stack validation optimisation. ([#3387](#3387)) ### Features * Base rollup in noir ([#3257](#3257)) ([4a1e9c3](4a1e9c3)) * Call stack validation optimisation. ([#3387](#3387)) ([d06d5db](d06d5db)) * Goblin proof construction ([#3332](#3332)) ([6a7ebb6](6a7ebb6)) * More logs relevant for debugging failures of 2 pixies test ([#3370](#3370)) ([683a0f3](683a0f3)) * Noir subrepo. ([#3369](#3369)) ([d94d88b](d94d88b)) * Noir_wasm compilation of noir programs ([#3272](#3272)) ([f9981d5](f9981d5)) * Rollback public state changes on failure ([#3393](#3393)) ([0e276fb](0e276fb)) ### Bug Fixes * **docs:** Doc explaining noir debug_log ([#3322](#3322)) ([eed023d](eed023d)) * Naming inconsistency in private kernel ([#3384](#3384)) ([4743486](4743486)) * Race condition in `PXE.getTxReceipt(...)` ([#3411](#3411)) ([9557a66](9557a66)) ### Miscellaneous * Deprecate circuits/cpp ([#3421](#3421)) ([4973cfb](4973cfb)) * Deterministically deduplicate `cached_partial_non_native_field_multiplication` across wasm32 and native compilations ([#3425](#3425)) ([5524933](5524933)) * **docs:** Common patterns and anti patterns in aztec.nr ([#3413](#3413)) ([65bd855](65bd855)) * Fix and reenable e2e quick start ([#3403](#3403)) ([112740e](112740e)), closes [#3356](#3356) * Fix intermittent failures for block-building e2e test ([#3404](#3404)) ([e76e2d4](e76e2d4)), closes [#3358](#3358) * Formatted `noir-contracts` and `aztec-nr` ([a73c4aa](a73c4aa)) * Initial clone of noir to subrepo ([#3409](#3409)) ([8f1cb83](8f1cb83)) * **noir-contracts:** Remove redundant return value of 1 ([#3415](#3415)) ([2001d47](2001d47)), closes [#2615](#2615) * Plumbs noir subrepo into yarn-project. ([#3420](#3420)) ([63173c4](63173c4)) * Remove pxe / node /p2p-bootstrap docker images ([#3396](#3396)) ([c236143](c236143)) * Skip artifacts for prettier ([#3399](#3399)) ([98d9e04](98d9e04)) * Update path to acir artifacts ([#3426](#3426)) ([f56f88d](f56f88d)) </details> <details><summary>barretenberg.js: 0.16.0</summary> ## [0.16.0](barretenberg.js-v0.15.1...barretenberg.js-v0.16.0) (2023-11-27) ### Miscellaneous * Plumbs noir subrepo into yarn-project. ([#3420](#3420)) ([63173c4](63173c4)) </details> <details><summary>barretenberg: 0.16.0</summary> ## [0.16.0](barretenberg-v0.15.1...barretenberg-v0.16.0) (2023-11-27) ### Features * Goblin proof construction ([#3332](#3332)) ([6a7ebb6](6a7ebb6)) * Noir subrepo. ([#3369](#3369)) ([d94d88b](d94d88b)) ### Miscellaneous * Deterministically deduplicate `cached_partial_non_native_field_multiplication` across wasm32 and native compilations ([#3425](#3425)) ([5524933](5524933)) * Plumbs noir subrepo into yarn-project. ([#3420](#3420)) ([63173c4](63173c4)) * Update path to acir artifacts ([#3426](#3426)) ([f56f88d](f56f88d)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.16.0</summary> ## [0.16.0](AztecProtocol/aztec-packages@aztec-packages-v0.15.1...aztec-packages-v0.16.0) (2023-11-27) ### ⚠ BREAKING CHANGES * deprecate circuits/cpp ([#3421](AztecProtocol/aztec-packages#3421)) * call stack validation optimisation. ([#3387](AztecProtocol/aztec-packages#3387)) ### Features * Base rollup in noir ([#3257](AztecProtocol/aztec-packages#3257)) ([4a1e9c3](AztecProtocol/aztec-packages@4a1e9c3)) * Call stack validation optimisation. ([#3387](AztecProtocol/aztec-packages#3387)) ([d06d5db](AztecProtocol/aztec-packages@d06d5db)) * Goblin proof construction ([#3332](AztecProtocol/aztec-packages#3332)) ([6a7ebb6](AztecProtocol/aztec-packages@6a7ebb6)) * More logs relevant for debugging failures of 2 pixies test ([#3370](AztecProtocol/aztec-packages#3370)) ([683a0f3](AztecProtocol/aztec-packages@683a0f3)) * Noir subrepo. ([#3369](AztecProtocol/aztec-packages#3369)) ([d94d88b](AztecProtocol/aztec-packages@d94d88b)) * Noir_wasm compilation of noir programs ([#3272](AztecProtocol/aztec-packages#3272)) ([f9981d5](AztecProtocol/aztec-packages@f9981d5)) * Rollback public state changes on failure ([#3393](AztecProtocol/aztec-packages#3393)) ([0e276fb](AztecProtocol/aztec-packages@0e276fb)) ### Bug Fixes * **docs:** Doc explaining noir debug_log ([#3322](AztecProtocol/aztec-packages#3322)) ([eed023d](AztecProtocol/aztec-packages@eed023d)) * Naming inconsistency in private kernel ([#3384](AztecProtocol/aztec-packages#3384)) ([4743486](AztecProtocol/aztec-packages@4743486)) * Race condition in `PXE.getTxReceipt(...)` ([#3411](AztecProtocol/aztec-packages#3411)) ([9557a66](AztecProtocol/aztec-packages@9557a66)) ### Miscellaneous * Deprecate circuits/cpp ([#3421](AztecProtocol/aztec-packages#3421)) ([4973cfb](AztecProtocol/aztec-packages@4973cfb)) * Deterministically deduplicate `cached_partial_non_native_field_multiplication` across wasm32 and native compilations ([#3425](AztecProtocol/aztec-packages#3425)) ([5524933](AztecProtocol/aztec-packages@5524933)) * **docs:** Common patterns and anti patterns in aztec.nr ([#3413](AztecProtocol/aztec-packages#3413)) ([65bd855](AztecProtocol/aztec-packages@65bd855)) * Fix and reenable e2e quick start ([#3403](AztecProtocol/aztec-packages#3403)) ([112740e](AztecProtocol/aztec-packages@112740e)), closes [#3356](AztecProtocol/aztec-packages#3356) * Fix intermittent failures for block-building e2e test ([#3404](AztecProtocol/aztec-packages#3404)) ([e76e2d4](AztecProtocol/aztec-packages@e76e2d4)), closes [#3358](AztecProtocol/aztec-packages#3358) * Formatted `noir-contracts` and `aztec-nr` ([a73c4aa](AztecProtocol/aztec-packages@a73c4aa)) * Initial clone of noir to subrepo ([#3409](AztecProtocol/aztec-packages#3409)) ([8f1cb83](AztecProtocol/aztec-packages@8f1cb83)) * **noir-contracts:** Remove redundant return value of 1 ([#3415](AztecProtocol/aztec-packages#3415)) ([2001d47](AztecProtocol/aztec-packages@2001d47)), closes [#2615](AztecProtocol/aztec-packages#2615) * Plumbs noir subrepo into yarn-project. ([#3420](AztecProtocol/aztec-packages#3420)) ([63173c4](AztecProtocol/aztec-packages@63173c4)) * Remove pxe / node /p2p-bootstrap docker images ([#3396](AztecProtocol/aztec-packages#3396)) ([c236143](AztecProtocol/aztec-packages@c236143)) * Skip artifacts for prettier ([#3399](AztecProtocol/aztec-packages#3399)) ([98d9e04](AztecProtocol/aztec-packages@98d9e04)) * Update path to acir artifacts ([#3426](AztecProtocol/aztec-packages#3426)) ([f56f88d](AztecProtocol/aztec-packages@f56f88d)) </details> <details><summary>barretenberg.js: 0.16.0</summary> ## [0.16.0](AztecProtocol/aztec-packages@barretenberg.js-v0.15.1...barretenberg.js-v0.16.0) (2023-11-27) ### Miscellaneous * Plumbs noir subrepo into yarn-project. ([#3420](AztecProtocol/aztec-packages#3420)) ([63173c4](AztecProtocol/aztec-packages@63173c4)) </details> <details><summary>barretenberg: 0.16.0</summary> ## [0.16.0](AztecProtocol/aztec-packages@barretenberg-v0.15.1...barretenberg-v0.16.0) (2023-11-27) ### Features * Goblin proof construction ([#3332](AztecProtocol/aztec-packages#3332)) ([6a7ebb6](AztecProtocol/aztec-packages@6a7ebb6)) * Noir subrepo. ([#3369](AztecProtocol/aztec-packages#3369)) ([d94d88b](AztecProtocol/aztec-packages@d94d88b)) ### Miscellaneous * Deterministically deduplicate `cached_partial_non_native_field_multiplication` across wasm32 and native compilations ([#3425](AztecProtocol/aztec-packages#3425)) ([5524933](AztecProtocol/aztec-packages@5524933)) * Plumbs noir subrepo into yarn-project. ([#3420](AztecProtocol/aztec-packages#3420)) ([63173c4](AztecProtocol/aztec-packages@63173c4)) * Update path to acir artifacts ([#3426](AztecProtocol/aztec-packages#3426)) ([f56f88d](AztecProtocol/aztec-packages@f56f88d)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

charlielye added 2 commits November 20, 2023 19:22

Build noir.

b5307cc

charlielye marked this pull request as draft November 20, 2023 20:47

charlielye added 26 commits November 21, 2023 17:14

Arm support.

ebda6cd

[ci rebuild noir-x86_64] [ci rebuild noir-arm64]

7892094

[ci rebuild noir]

4777acb

[ci rebuild noir]

ac89576

[ci rebuild noir]

637b4f6

[ci rebuild noir]

0cbeb36

[ci rebuild noir]

6754f48

[ci rebuild noir]

73c5397

[ci rebuild noir]

675d2f5

git subrepo pull (merge) noir

306ff3a

subrepo: subdir: "noir" merged: "816deeb60" upstream: origin: "[email protected]:noir-lang/noir" branch: "master" commit: "63d414c06" git-subrepo: version: "0.4.6" origin: "https://github.com/ingydotnet/git-subrepo" commit: "110b9eb"

[ci rebuild noir]

631f45b

Noir deploy

ee47b09

git subrepo commit (merge) noir

1661475

subrepo: subdir: "noir" merged: "00af10514" upstream: origin: "[email protected]:noir-lang/noir" branch: "master" commit: "251b930de" git-subrepo: version: "0.4.6" origin: "https://github.com/ingydotnet/git-subrepo" commit: "110b9eb"

Move to machine class to get docker

74d9048

[ci rebuild noir]

c13e89d

Alpine release build of noir.

b09c67c

Remove all the ARCH hacks from build system. Simplify deploy_dockerhub.

61999e7

Remove unused save log stuff.

1ab0800

Fixes. Can trigger dry deploy via cci.

601068e

Try again. Now there is workflow distinction for deploy, but we early…

6d60019

… out, unless dry deploy is set.

fix

fb85b60

fix

a5113c5

fix

8989ca1

fix

edefa32

fix

92379d6

fix

7f54df4

charlielye added 10 commits November 23, 2023 10:10

fix

cecc803

fix

bdbbb96

fix

1eb9dba

fix

a77703a

[ci dry-deploy]

b61a161

[ci dry-deploy]

edad1fc

Merge remote-tracking branch 'origin/master' into cl/noir-subrepo

1777aab

Merge branch 'master' into cl/noir-subrepo

ac19f40

fix

ca4ffa5

[ci dry-deploy]

b563913

charlielye marked this pull request as ready for review November 23, 2023 12:56

charlielye requested review from ludamad, Maddiaa0 and spypsy and removed request for ludamad November 23, 2023 12:56

fix [ci dry-deploy]

8e4867b

spypsy reviewed Nov 23, 2023

View reviewed changes

fix [ci dry-deploy]

96566b1

spypsy reviewed Nov 23, 2023

View reviewed changes

spypsy approved these changes Nov 23, 2023

View reviewed changes

Maddiaa0 approved these changes Nov 23, 2023

View reviewed changes

Address review comments. [ci rebuild]

2e6135b

Maddiaa0 approved these changes Nov 23, 2023

View reviewed changes

charlielye enabled auto-merge (squash) November 23, 2023 15:46

charlielye merged commit d94d88b into master Nov 23, 2023
82 checks passed

charlielye deleted the cl/noir-subrepo branch November 23, 2023 16:02

AztecBot mentioned this pull request Nov 23, 2023

chore(master): Release 0.16.0 #3385

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Noir subrepo. #3369

feat: Noir subrepo. #3369

charlielye commented Nov 20, 2023 •

edited

Loading

ludamad commented Nov 20, 2023

spypsy Nov 23, 2023

charlielye Nov 23, 2023 •

edited

Loading

spypsy Nov 23, 2023

charlielye Nov 23, 2023

charlielye Nov 23, 2023

Maddiaa0 Nov 23, 2023

feat: Noir subrepo. #3369

feat: Noir subrepo. #3369

Conversation

charlielye commented Nov 20, 2023 • edited Loading

ludamad commented Nov 20, 2023

spypsy Nov 23, 2023

Choose a reason for hiding this comment

charlielye Nov 23, 2023 • edited Loading

Choose a reason for hiding this comment

spypsy Nov 23, 2023

Choose a reason for hiding this comment

charlielye Nov 23, 2023

Choose a reason for hiding this comment

charlielye Nov 23, 2023

Choose a reason for hiding this comment

Maddiaa0 Nov 23, 2023

Choose a reason for hiding this comment

charlielye commented Nov 20, 2023 •

edited

Loading

charlielye Nov 23, 2023 •

edited

Loading