Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add dependency resolver for noir_wasm and implement FileManager for consistency with native interface #3891

Merged
merged 53 commits into from
Jan 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
1826d5d
Started merging changes
Thunkar Dec 18, 2023
29c1dda
Moving tests to jest
Thunkar Dec 18, 2023
d40c1ef
Fixed CJS compilation
Thunkar Dec 18, 2023
c857489
Merge branch 'master' of github.com:noir-lang/noir into gj/noir_wasm
Thunkar Dec 18, 2023
9129dbd
Cleanup
Thunkar Dec 19, 2023
487d48e
Working, tests green
Thunkar Dec 20, 2023
3cf9f97
Merge branch 'master' of github.com:noir-lang/noir into gj/noir_wasm
Thunkar Dec 20, 2023
89ef603
Merge branch 'master' of github.com:noir-lang/noir into gj/noir_wasm
Thunkar Dec 21, 2023
c3233a5
Deps cleanup
Thunkar Dec 21, 2023
81c7978
chore: try to build noir_wasm with yarn
sirasistant Dec 21, 2023
6335dc8
chore: build the fixtures on test
sirasistant Dec 21, 2023
edbe2fb
chore: add nargo to path
sirasistant Dec 21, 2023
c29145b
Fix types
Thunkar Dec 21, 2023
777d590
Used require for cjs import
Thunkar Dec 21, 2023
6efba0c
Moved integration tests to the correct package
Thunkar Dec 21, 2023
f35632e
Merge branch 'master' of github.com:noir-lang/noir into gj/noir_wasm
Thunkar Dec 21, 2023
3a0c688
Updated lock
Thunkar Dec 21, 2023
f983373
Fixed test loader
Thunkar Dec 21, 2023
4eb60f0
Fixed lock, module resolution
Thunkar Dec 21, 2023
9ebe032
Merge branch 'master' into gj/noir_wasm
Thunkar Dec 21, 2023
23f8b6e
chore: renames and removal of duplicated types
sirasistant Dec 22, 2023
e3b61ff
chore: zip dependency in test
sirasistant Dec 22, 2023
e2ec562
Merge branch 'master' into gj/noir_wasm
Thunkar Jan 2, 2024
d9b8200
Merge branch 'master' of github.com:noir-lang/noir into gj/noir_wasm
Thunkar Jan 2, 2024
64c8364
Changed compilation for better bundling
Thunkar Jan 3, 2024
61d8cf6
Merge branch 'master' into gj/noir_wasm
Thunkar Jan 5, 2024
e3156b8
logs
Thunkar Jan 5, 2024
b535ec5
Increased timeout
Thunkar Jan 5, 2024
f808f94
removed traces
Thunkar Jan 5, 2024
11fa714
Addressed PR feedback
Thunkar Jan 8, 2024
5cb08f3
Removed vscode settings
Thunkar Jan 8, 2024
a962f03
Merge branch 'master' of github.com:noir-lang/noir into gj/noir_wasm
Thunkar Jan 8, 2024
f6769b2
Merge branch 'master' of github.com:noir-lang/noir into gj/noir_wasm
Thunkar Jan 8, 2024
3cfdaad
Addressed feedback
Thunkar Jan 8, 2024
766440d
More PR comments
Thunkar Jan 8, 2024
dd7f8f4
Changed cache key
Thunkar Jan 8, 2024
e87e9bc
Merge branch 'master' of github.com:noir-lang/noir into gj/noir_wasm
Thunkar Jan 8, 2024
f0339d5
oops typo
Thunkar Jan 8, 2024
a9664df
Increased timeouts
Thunkar Jan 8, 2024
c51581e
Removed paths from tsconfig.json
Thunkar Jan 8, 2024
83cb722
Merge branch 'master' of github.com:noir-lang/noir into gj/noir_wasm
Thunkar Jan 8, 2024
cd9bb3c
Removed debug flag
Thunkar Jan 8, 2024
ca744ae
Moved timeouts to tests and not suites
Thunkar Jan 8, 2024
c8b5aa6
Make wasm-pack mimic the opt behavior of other packages
Thunkar Jan 9, 2024
8935bf0
Merge branch 'master' into gj/noir_wasm
Thunkar Jan 9, 2024
de08e5e
Update compiler/integration-tests/scripts/codegen-verifiers.sh
Thunkar Jan 9, 2024
3010201
allow embedding stdlib + wasm-opt disable
Thunkar Jan 9, 2024
c8bd920
Merge branch 'gj/noir_wasm' of github.com:noir-lang/noir into gj/noir…
Thunkar Jan 9, 2024
7115d28
fixed unused dep warning
Thunkar Jan 9, 2024
0c9e195
removed @ts-ignore, proper typings
Thunkar Jan 9, 2024
002ae8a
Merge branch 'master' of github.com:noir-lang/noir into gj/noir_wasm
Thunkar Jan 9, 2024
c15d48a
fixed compilation after clean
Thunkar Jan 9, 2024
f7f7f5e
Merge branch 'master' into gj/noir_wasm
TomAFrench Jan 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 28 additions & 22 deletions .github/workflows/test-js-packages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,35 +47,42 @@ jobs:
retention-days: 3

build-noir-wasm:
needs: [build-noirc-abi]
runs-on: ubuntu-latest
timeout-minutes: 30

steps:
- name: Checkout sources
uses: actions/checkout@v4

- name: Setup Nix
uses: ./.github/actions/nix
- name: Setup toolchain
uses: dtolnay/[email protected]

- uses: Swatinem/rust-cache@v2
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
nix-cache-name: "noir"
cachix-auth-token: ${{ secrets.CACHIXAUTHTOKEN }}
key: noir-wasm
cache-on-failure: true
save-if: ${{ github.event_name != 'merge_group' }}

- name: Build wasm package
run: |
nix build -L .#noir_wasm
- name: Download noirc_abi_wasm package artifact
uses: actions/download-artifact@v3
with:
name: noirc_abi_wasm
path: ./tooling/noirc_abi_wasm

- name: Dereference symlink
run: echo "UPLOAD_PATH=$(readlink -f ./result/noir_wasm)" >> $GITHUB_ENV
- name: Install Yarn dependencies
uses: ./.github/actions/setup

- name: Build noir_wasm
run: yarn workspace @noir-lang/noir_wasm build

- name: Upload artifact
uses: actions/upload-artifact@v3
with:
name: noir_wasm
path: ${{ env.UPLOAD_PATH }}
path: ./compiler/wasm
retention-days: 3


build-acvm-js:
runs-on: ubuntu-latest
timeout-minutes: 30
Expand Down Expand Up @@ -289,25 +296,24 @@ jobs:
name: noir_wasm
path: ./compiler/wasm

- name: Install Yarn dependencies
uses: ./.github/actions/setup

- name: Download nargo binary
uses: actions/download-artifact@v3
with:
name: nargo
path: ./nargo

- name: Compile fixtures with Nargo CLI
working-directory: ./compiler/wasm/fixtures
- name: Set nargo on PATH
run: |
nargo_binary=${{ github.workspace }}/nargo/nargo
nargo_binary="${{ github.workspace }}/nargo/nargo"
chmod +x $nargo_binary
for dir in $(ls -d */); do
pushd $dir/noir-script
$nargo_binary compile
popd
done
echo "$(dirname $nargo_binary)" >> $GITHUB_PATH
export PATH="$PATH:$(dirname $nargo_binary)"

- name: Install Yarn dependencies
uses: ./.github/actions/setup
- name: Build fixtures
run: yarn workspace @noir-lang/noir_wasm test:build_fixtures

- name: Install Playwright
uses: ./.github/actions/install-playwright
Expand Down
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ jsonrpc = { version = "0.16.0", features = ["minreq_http"] }
tracing = "0.1.40"
tracing-web = "0.1.3"
tracing-subscriber = { version = "0.3.18", features = ["env-filter"] }
rust-embed = "6.6.0"

[profile.dev]
# This is required to be able to run `cargo test` in acvm_js due to the `locals exceeds maximum` error.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,22 +1,13 @@
import { expect } from '@esm-bundle/chai';
import * as TOML from 'smol-toml';

import newCompiler, {
CompiledProgram,
PathToFileSourceMap,
compile,
init_log_level as compilerLogLevel,
} from '@noir-lang/noir_wasm';
import { compile, createFileManager } from '@noir-lang/noir_wasm';
import { Noir } from '@noir-lang/noir_js';
import { InputMap } from '@noir-lang/noirc_abi';
import { BarretenbergBackend } from '@noir-lang/backend_barretenberg';

import { getFile } from './utils.js';

await newCompiler();

compilerLogLevel('INFO');

const test_cases = [
{
case: 'test_programs/execution_success/1_mul',
Expand All @@ -32,12 +23,11 @@ const suite = Mocha.Suite.create(mocha.suite, 'Noir end to end test');

suite.timeout(60 * 20e3); //20mins

function getCircuit(noirSource: string): CompiledProgram {
const sourceMap = new PathToFileSourceMap();
sourceMap.add_source_code('main.nr', noirSource);

// We're ignoring this in the resolver but pass in something sensible.
const result = compile('main.nr', undefined, undefined, sourceMap);
async function getCircuit(projectPath: string) {
const fm = createFileManager('/');
await fm.writeFile('./src/main.nr', await getFile(`${projectPath}/src/main.nr`));
await fm.writeFile('./Nargo.toml', await getFile(`${projectPath}/Nargo.toml`));
const result = await compile(fm);
if (!('program' in result)) {
throw new Error('Compilation failed');
}
Expand All @@ -51,11 +41,9 @@ test_cases.forEach((testInfo) => {
const base_relative_path = '../../../../..';
const test_case = testInfo.case;

const noir_source = await getFile(`${base_relative_path}/${test_case}/src/main.nr`);

let noir_program: CompiledProgram;
let noir_program;
try {
noir_program = getCircuit(noir_source);
noir_program = await getCircuit(`${base_relative_path}/${test_case}`);

expect(noir_program, 'Compile output ').to.be.an('object');
} catch (e) {
Expand All @@ -66,7 +54,7 @@ test_cases.forEach((testInfo) => {
const backend = new BarretenbergBackend(noir_program);
const program = new Noir(noir_program, backend);

const prover_toml = await getFile(`${base_relative_path}/${test_case}/Prover.toml`);
const prover_toml = await new Response(await getFile(`${base_relative_path}/${test_case}/Prover.toml`)).text();
const inputs: InputMap = TOML.parse(prover_toml) as InputMap;

// JS Proving
Expand Down
31 changes: 9 additions & 22 deletions compiler/integration-tests/test/browser/recursion.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,59 +2,46 @@
import { expect } from '@esm-bundle/chai';
import { TEST_LOG_LEVEL } from '../environment.js';
import { Logger } from 'tslog';
import newCompiler, {
CompiledProgram,
PathToFileSourceMap,
compile,
init_log_level as compilerLogLevel,
} from '@noir-lang/noir_wasm';
import { acvm, abi, Noir } from '@noir-lang/noir_js';

import * as TOML from 'smol-toml';
import { BarretenbergBackend } from '@noir-lang/backend_barretenberg';
import { getFile } from './utils.js';
import { Field, InputMap } from '@noir-lang/noirc_abi';
import { createFileManager, compile } from '@noir-lang/noir_wasm';

const logger = new Logger({ name: 'test', minLevel: TEST_LOG_LEVEL });

const { default: initACVM } = acvm;
const { default: newABICoder } = abi;

await newCompiler();
await newABICoder();
await initACVM();

compilerLogLevel('INFO');

const base_relative_path = '../../../../..';
const circuit_main = 'test_programs/execution_success/assert_statement';
const circuit_recursion = 'compiler/integration-tests/circuits/recursion';

function getCircuit(noirSource: string): CompiledProgram {
const sourceMap = new PathToFileSourceMap();
sourceMap.add_source_code('main.nr', noirSource);
const result = compile('main.nr', undefined, undefined, sourceMap);
async function getCircuit(projectPath: string) {
const fm = createFileManager('/');
await fm.writeFile('./src/main.nr', await getFile(`${projectPath}/src/main.nr`));
await fm.writeFile('./Nargo.toml', await getFile(`${projectPath}/Nargo.toml`));
const result = await compile(fm);
if (!('program' in result)) {
throw new Error('Compilation failed');
}

return result.program;
}

describe('It compiles noir program code, receiving circuit bytes and abi object.', () => {
let circuit_main_source;
let circuit_main_toml;
let circuit_recursion_source;

before(async () => {
circuit_main_source = await getFile(`${base_relative_path}/${circuit_main}/src/main.nr`);
circuit_main_toml = await getFile(`${base_relative_path}/${circuit_main}/Prover.toml`);

circuit_recursion_source = await getFile(`${base_relative_path}/${circuit_recursion}/src/main.nr`);
circuit_main_toml = await new Response(await getFile(`${base_relative_path}/${circuit_main}/Prover.toml`)).text();
});

it('Should generate valid inner proof for correct input, then verify proof within a proof', async () => {
const main_program = getCircuit(circuit_main_source);
const main_program = await getCircuit(`${base_relative_path}/${circuit_main}`);
const main_inputs: InputMap = TOML.parse(circuit_main_toml) as InputMap;

const main_backend = new BarretenbergBackend(main_program);
Expand Down Expand Up @@ -83,7 +70,7 @@ describe('It compiles noir program code, receiving circuit bytes and abi object.

logger.debug('recursion_inputs', recursion_inputs);

const recursion_program = await getCircuit(circuit_recursion_source);
const recursion_program = await getCircuit(`${base_relative_path}/${circuit_recursion}`);

const recursion_backend = new BarretenbergBackend(recursion_program);

Expand Down
4 changes: 2 additions & 2 deletions compiler/integration-tests/test/browser/utils.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
export async function getFile(file_path: string): Promise<string> {
export async function getFile(file_path: string): Promise<ReadableStream<Uint8Array>> {
const file_url = new URL(file_path, import.meta.url);
const response = await fetch(file_url);

if (!response.ok) throw new Error('Network response was not OK');

return await response.text();
return response.body as ReadableStream<Uint8Array>;
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,44 +2,45 @@ import { expect } from 'chai';
import { ethers } from 'hardhat';

import { readFileSync } from 'node:fs';
import { resolve } from 'path';
import { resolve, join } from 'path';
import toml from 'toml';

import {
compile,
CompiledProgram,
init_log_level as compilerLogLevel,
PathToFileSourceMap,
} from '@noir-lang/noir_wasm';
import { Noir } from '@noir-lang/noir_js';
import { BarretenbergBackend } from '@noir-lang/backend_barretenberg';
import { Field, InputMap } from '@noir-lang/noirc_abi';

compilerLogLevel('INFO');
import { compile, createFileManager } from '@noir-lang/noir_wasm';

it(`smart contract can verify a recursive proof`, async () => {
const innerSourcePath = resolve(`../../test_programs/execution_success/assert_statement/src/main.nr`);
const sourceMapInnerProgram = new PathToFileSourceMap();
sourceMapInnerProgram.add_source_code(innerSourcePath, readFileSync(innerSourcePath, 'utf-8'));
const innerProgram = (
compile(innerSourcePath, undefined, undefined, sourceMapInnerProgram) as { program: CompiledProgram }
).program;

const recursionSourcePath = resolve(`./circuits/recursion/src/main.nr`);
const sourceMapRecursionProgram = new PathToFileSourceMap();
sourceMapRecursionProgram.add_source_code(recursionSourcePath, readFileSync(recursionSourcePath, 'utf-8'));
const recursionProgram = (
compile(recursionSourcePath, undefined, undefined, sourceMapRecursionProgram) as { program: CompiledProgram }
).program;
const basePath = resolve(join(__dirname, '../../../../'));
const fm = createFileManager(basePath);
const innerCompilationResult = await compile(
fm,
join(basePath, './test_programs/execution_success/assert_statement'),
);
if (!('program' in innerCompilationResult)) {
throw new Error('Compilation failed');
}
const innerProgram = innerCompilationResult.program;

const recursionCompilationResult = await compile(
fm,
join(basePath, './compiler/integration-tests/circuits/recursion'),
);
if (!('program' in recursionCompilationResult)) {
throw new Error('Compilation failed');
}
const recursionProgram = recursionCompilationResult.program;

// Intermediate proof

const inner_backend = new BarretenbergBackend(innerProgram);
const inner = new Noir(innerProgram);

const inner_prover_toml = readFileSync(
resolve(`../../test_programs/execution_success/assert_statement/Prover.toml`),
join(basePath, `./test_programs/execution_success/assert_statement/Prover.toml`),
).toString();

const inner_inputs = toml.parse(inner_prover_toml);

const { witness: main_witness } = await inner.execute(inner_inputs);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,10 @@ import { readFileSync } from 'node:fs';
import { resolve } from 'path';
import toml from 'toml';

import { PathToFileSourceMap, compile, init_log_level as compilerLogLevel } from '@noir-lang/noir_wasm';
import { Noir } from '@noir-lang/noir_js';
import { BarretenbergBackend } from '@noir-lang/backend_barretenberg';

compilerLogLevel('INFO');
import { compile, createFileManager } from '@noir-lang/noir_wasm';

const test_cases = [
{
Expand All @@ -31,11 +30,8 @@ test_cases.forEach((testInfo) => {
const base_relative_path = '../..';
const test_case = testInfo.case;

const noirSourcePath = resolve(`${base_relative_path}/${test_case}/src/main.nr`);
const sourceMap = new PathToFileSourceMap();
sourceMap.add_source_code(noirSourcePath, readFileSync(noirSourcePath, 'utf-8'));

const compileResult = compile(noirSourcePath, undefined, undefined, sourceMap);
const fm = createFileManager(resolve(`${base_relative_path}/${test_case}`));
const compileResult = await compile(fm);
if (!('program' in compileResult)) {
throw new Error('Compilation failed');
}
Expand Down
2 changes: 1 addition & 1 deletion compiler/noirc_driver/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ iter-extended.workspace = true
fm.workspace = true
serde.workspace = true
fxhash.workspace = true
rust-embed = "6.6.0"
rust-embed.workspace = true
tracing.workspace = true

aztec_macros = { path = "../../aztec_macros" }
2 changes: 1 addition & 1 deletion compiler/wasm/.eslintrc.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
module.exports = {
extends: ["../../.eslintrc.js"],
extends: ['../../.eslintrc.js'],
};
2 changes: 2 additions & 0 deletions compiler/wasm/.gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
noir-script/target
dist
build
18 changes: 12 additions & 6 deletions compiler/wasm/.mocharc.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
{
"extension": [
"ts"
],
"spec": "test/node/**/*.test.ts",
"require": "ts-node/register"
}
"require": "ts-node/register",
"extensions": [
"ts"
],
"spec": [
"./test/**/!(browser)/*.test.ts"
],
"node-option": [
"loader=ts-node"
]
}

Loading
Loading