Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: noir_wasm compilation of noir programs #3272

Merged
merged 42 commits into from
Nov 22, 2023
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
41413c5
stop committing artifacts
dan-aztec Nov 6, 2023
b2c5741
undo box artifact changes
dan-aztec Nov 6, 2023
9abd0ff
compile non-contract package types with noir-wasm-compiler.ts
dan-aztec Nov 6, 2023
b51c19a
revert "binary" to "bin"
dan-aztec Nov 6, 2023
e44955d
bin
dan-aztec Nov 6, 2023
fa43b73
revert to nargo for compiler
dan-aztec Nov 6, 2023
27c62b2
use types, still broken
dan-aztec Nov 6, 2023
ae7ef52
remove src/target from git
dan-aztec Nov 6, 2023
94d6399
handle non-contract return type
dan-aztec Nov 6, 2023
1473663
generate types
dan-aztec Nov 6, 2023
3a9bd7e
handle program differently from contract
dan-aztec Nov 7, 2023
d62887e
handle program and contract separately
dan-aztec Nov 7, 2023
cde221e
update test
dan-aztec Nov 7, 2023
d26d6ac
stop logging
dan-aztec Nov 7, 2023
d1b1d4e
fix type
dan-aztec Nov 7, 2023
c34bb69
switch command name to compile
dan-aztec Nov 7, 2023
c79086e
readd src/target for testing
dan-aztec Nov 7, 2023
d1f1ecb
point outdir to src/target
dan-aztec Nov 7, 2023
e655751
single build, and absoltue path to /src/target
dan-aztec Nov 7, 2023
7ebbf5d
merge conflicts
dan-aztec Nov 8, 2023
452b858
checkout origin/master barretenberg?
dan-aztec Nov 8, 2023
c635a05
fetch
dan-aztec Nov 8, 2023
e2c70da
Revert "checkout origin/master barretenberg?"
dan-aztec Nov 8, 2023
eae79ec
revert noir-protocol-circuits changes
dan-aztec Nov 8, 2023
cf82db1
origin not local
dan-aztec Nov 8, 2023
1f38e39
revert more protocol circuits changes
dan-aztec Nov 8, 2023
71ba52f
rename to compileNoir
dan-aztec Nov 14, 2023
0911592
Merge branch 'master' into dan/build-artifacts-ci-merge-conflicgt
dan-aztec Nov 14, 2023
955417e
update yarn command to compile
dan-aztec Nov 14, 2023
8efa75a
revert formatting
dan-aztec Nov 14, 2023
47b8c2b
Merge branch 'master' into dan/build-artifacts-ci-merge-conflicgt
dan-aztec Nov 16, 2023
af5832f
update compileContract name
dan-aztec Nov 16, 2023
eb6f46e
Merge branch 'master' into dan/build-artifacts-ci-merge-conflicgt
dan-aztec Nov 17, 2023
d7abdb5
merge conflict
dan-aztec Nov 18, 2023
c2659eb
small change to output json data
dan-aztec Nov 18, 2023
857101e
move generateTypescript logic out of noir-protocol-circuits
dan-aztec Nov 18, 2023
d3a51cc
write program types to ../types/
dan-aztec Nov 18, 2023
2dde02d
remove formatted deleted code
dan-aztec Nov 21, 2023
d2af58d
revert cpp format
dan-aztec Nov 21, 2023
d5ada69
revert package.json change
dan-aztec Nov 21, 2023
ac66988
formatting
dan-aztec Nov 21, 2023
b0be295
formatting
dan-aztec Nov 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@ using GetParameterView = std::conditional_t<IsField<typename Params::DataType>,

template <typename T, size_t subrelation_idx>
concept HasSubrelationLinearlyIndependentMember = requires(T) {
{
std::get<subrelation_idx>(T::SUBRELATION_LINEARLY_INDEPENDENT)
} -> std::convertible_to<bool>;
};
{
std::get<subrelation_idx>(T::SUBRELATION_LINEARLY_INDEPENDENT)
} -> std::convertible_to<bool>;
};

template <typename T>
concept HasParameterLengthAdjustmentsMember = requires { T::TOTAL_LENGTH_ADJUSTMENTS; };
Expand Down
3 changes: 1 addition & 2 deletions barretenberg/cpp/src/barretenberg/relations/utils.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,7 @@ template <typename Flavor> class RelationUtils {
template <typename... T>
static constexpr void add_tuples(std::tuple<T...>& tuple_1, const std::tuple<T...>& tuple_2)
{
auto add_tuples_helper = [&]<std::size_t... I>(std::index_sequence<I...>)
{
auto add_tuples_helper = [&]<std::size_t... I>(std::index_sequence<I...>) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

must have an auto formatter on, can revert

((std::get<I>(tuple_1) += std::get<I>(tuple_2)), ...);
};

Expand Down
2 changes: 1 addition & 1 deletion yarn-project/noir-compiler/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ const log = createConsoleLogger('aztec:compiler-cli');

const main = async () => {
program.name('aztec-compile');
compileContract(program, 'contract', log);
compileContract(program, 'compile', log);
generateTypescriptInterface(program, 'typescript', log);
generateNoirInterface(program, 'interface', log);
await program.parseAsync(process.argv);
Expand Down
158 changes: 104 additions & 54 deletions yarn-project/noir-compiler/src/cli/contract.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { ContractArtifact } from '@aztec/foundation/abi';
import { LogFn } from '@aztec/foundation/log';

import { Command } from 'commander';
Expand All @@ -6,81 +7,130 @@ import { mkdirpSync } from 'fs-extra';
import path, { resolve } from 'path';

import {
ProgramArtifact,
compileUsingNargo,
compileUsingNoirWasm,
generateNoirContractInterface,
generateTypescriptContractInterface,
} from '../index.js';

/**
* CLI options for configuring behavior
*/
interface options {
// eslint-disable-next-line jsdoc/require-jsdoc
outdir: string;
// eslint-disable-next-line jsdoc/require-jsdoc
typescript: string | undefined;
// eslint-disable-next-line jsdoc/require-jsdoc
interface: string | undefined;
// eslint-disable-next-line jsdoc/require-jsdoc
compiler: string | undefined;
}
/**
* Registers a 'contract' command on the given commander program that compiles an Aztec.nr contract project.
* @param program - Commander program.
* @param log - Optional logging function.
* @returns The program with the command registered.
*/
export function compileContract(program: Command, name = 'contract', log: LogFn = () => {}): Command {
export function compileContract(program: Command, name = 'compile', log: LogFn = () => {}): Command {
return program
.command(name)
.argument('<project-path>', 'Path to the Aztec.nr project to compile')
.argument('<project-path>', 'Path to the bin or Aztec.nr project to compile')
.option('-o, --outdir <path>', 'Output folder for the binary artifacts, relative to the project path', 'target')
.option('-ts, --typescript <path>', 'Optional output folder for generating typescript wrappers', undefined)
.option('-i, --interface <path>', 'Optional output folder for generating an Aztec.nr contract interface', undefined)
.option('-c --compiler <string>', 'Which compiler to use. Either nargo or wasm. Defaults to nargo', 'wasm')
.description('Compiles the contracts in the target project')
.description('Compiles the Noir Source in the target project')

.action(
async (
projectPath: string,
/* eslint-disable jsdoc/require-jsdoc */
options: {
outdir: string;
typescript: string | undefined;
interface: string | undefined;
compiler: string | undefined;
},
/* eslint-enable jsdoc/require-jsdoc */
) => {
const { outdir, typescript, interface: noirInterface, compiler } = options;
if (typeof projectPath !== 'string') throw new Error(`Missing project path argument`);
if (compiler !== 'nargo' && compiler !== 'wasm') throw new Error(`Invalid compiler: ${compiler}`);
const currentDir = process.cwd();
.action(async (projectPath: string, options: options) => {
const { compiler } = options;
if (typeof projectPath !== 'string') throw new Error(`Missing project path argument`);
if (compiler !== 'nargo' && compiler !== 'wasm') throw new Error(`Invalid compiler: ${compiler}`);

const compile = compiler === 'wasm' ? compileUsingNoirWasm : compileUsingNargo;
log(`Compiling contracts...`);
const result = await compile(projectPath, { log });
const compile = compiler === 'wasm' ? compileUsingNoirWasm : compileUsingNargo;
log(`Compiling ${projectPath} with ${compiler} backend...`);
const results = await compile(projectPath, { log });
for (const result of results) {
generateOutput(projectPath, result, options, log);
}
});
}

for (const contract of result) {
const artifactPath = resolve(projectPath, outdir, `${contract.name}.json`);
log(`Writing ${contract.name} artifact to ${path.relative(currentDir, artifactPath)}`);
mkdirSync(path.dirname(artifactPath), { recursive: true });
writeFileSync(artifactPath, JSON.stringify(contract, null, 2));
/**
*
* @param contract - output from compiler, to serialize locally. branch based on Contract vs Program
*/
function generateOutput(
projectPath: string,
_result: ContractArtifact | ProgramArtifact,
options: options,
log: LogFn,
) {
const contract = _result as ContractArtifact;
if (contract.name) {
return generateContractOutput(projectPath, contract, options, log);
} else {
const program = _result as ProgramArtifact;
if (program.abi) {
return generateProgramOutput(projectPath, program, options, log);
}
}
}
/**
*
* @param program - output from compiler, to serialize locally
*/
function generateProgramOutput(projectPath: string, contract: ProgramArtifact, options: options, log: LogFn) {
const currentDir = process.cwd();
const { outdir, typescript, interface: noirInterface } = options;
const artifactPath = resolve(projectPath, outdir, `main.json`);
log(`Writing ${'main.json'} artifact to ${path.relative(currentDir, artifactPath)}`);
mkdirSync(path.dirname(artifactPath), { recursive: true });
writeFileSync(artifactPath, JSON.stringify(contract, null, 2));

if (noirInterface) {
// not implemented
}

if (typescript) {
// not implemented
}
}

/**
*
* @param contract - output from compiler, to serialize locally
*/
function generateContractOutput(projectPath: string, contract: ContractArtifact, options: options, log: LogFn) {
const currentDir = process.cwd();
const { outdir, typescript, interface: noirInterface } = options;
const artifactPath = resolve(projectPath, outdir, `${contract.name}.json`);
log(`Writing ${contract.name} artifact to ${path.relative(currentDir, artifactPath)}`);
mkdirSync(path.dirname(artifactPath), { recursive: true });
writeFileSync(artifactPath, JSON.stringify(contract, null, 2));

if (noirInterface) {
const noirInterfacePath = resolve(projectPath, noirInterface, `${contract.name}_interface.nr`);
log(
`Writing ${contract.name} Aztec.nr external interface to ${path.relative(currentDir, noirInterfacePath)}`,
);
const noirWrapper = generateNoirContractInterface(contract);
mkdirpSync(path.dirname(noirInterfacePath));
writeFileSync(noirInterfacePath, noirWrapper);
}
if (noirInterface) {
const noirInterfacePath = resolve(projectPath, noirInterface, `${contract.name}_interface.nr`);
log(`Writing ${contract.name} Aztec.nr external interface to ${path.relative(currentDir, noirInterfacePath)}`);
const noirWrapper = generateNoirContractInterface(contract);
mkdirpSync(path.dirname(noirInterfacePath));
writeFileSync(noirInterfacePath, noirWrapper);
}

if (typescript) {
const tsPath = resolve(projectPath, typescript, `${contract.name}.ts`);
log(`Writing ${contract.name} typescript interface to ${path.relative(currentDir, tsPath)}`);
let relativeArtifactPath = path.relative(path.dirname(tsPath), artifactPath);
log(`Relative path: ${relativeArtifactPath}`);
if (relativeArtifactPath === `${contract.name}.json`) {
// relative path edge case, prepending ./ for local import - the above logic just does
// `${contract.name}.json`, which is not a valid import for a file in the same directory
relativeArtifactPath = `./${contract.name}.json`;
}
log(`Relative path after correction: ${relativeArtifactPath}`);
const tsWrapper = generateTypescriptContractInterface(contract, relativeArtifactPath);
mkdirpSync(path.dirname(tsPath));
writeFileSync(tsPath, tsWrapper);
}
}
},
);
if (typescript) {
const tsPath = resolve(projectPath, typescript, `${contract.name}.ts`);
log(`Writing ${contract.name} typescript interface to ${path.relative(currentDir, tsPath)}`);
let relativeArtifactPath = path.relative(path.dirname(tsPath), artifactPath);
log(`Relative path: ${relativeArtifactPath}`);
if (relativeArtifactPath === `${contract.name}.json`) {
// relative path edge case, prepending ./ for local import - the above logic just does
// `${contract.name}.json`, which is not a valid import for a file in the same directory
relativeArtifactPath = `./${contract.name}.json`;
}
log(`Relative path after correction: ${relativeArtifactPath}`);
const tsWrapper = generateTypescriptContractInterface(contract, relativeArtifactPath);
mkdirpSync(path.dirname(tsPath));
writeFileSync(tsPath, tsWrapper);
}
}
6 changes: 3 additions & 3 deletions yarn-project/noir-compiler/src/compile/nargo.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import { emptyDirSync } from 'fs-extra';
import path from 'path';

import { NoirCommit, NoirTag } from '../index.js';
import { NoirCompilationArtifacts, NoirCompiledContract, NoirDebugMetadata } from '../noir_artifact.js';
import { NoirCompiledContract, NoirContractCompilationArtifacts, NoirDebugMetadata } from '../noir_artifact.js';

/** Compilation options */
export type CompileOpts = {
Expand All @@ -31,7 +31,7 @@ export class NargoContractCompiler {
* Compiles the contracts in projectPath and returns the Aztec.nr artifact.
* @returns Aztec.nr artifact of the compiled contracts.
*/
public compile(): Promise<NoirCompilationArtifacts[]> {
public compile(): Promise<NoirContractCompilationArtifacts[]> {
const stdio = this.opts.quiet ? 'ignore' : 'inherit';
const nargoBin = this.opts.nargoBin ?? 'nargo';
const version = execSync(`${nargoBin} --version`, { cwd: this.projectPath, stdio: 'pipe' }).toString();
Expand All @@ -51,7 +51,7 @@ export class NargoContractCompiler {
}
}

private collectArtifacts(): NoirCompilationArtifacts[] {
private collectArtifacts(): NoirContractCompilationArtifacts[] {
const contractArtifacts = new Map<string, NoirCompiledContract>();
const debugArtifacts = new Map<string, NoirDebugMetadata>();

Expand Down
68 changes: 58 additions & 10 deletions yarn-project/noir-compiler/src/compile/noir/noir-wasm-compiler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { LogFn, createDebugLogger } from '@aztec/foundation/log';
import { CompileError, compile } from '@noir-lang/noir_wasm';
import { isAbsolute } from 'node:path';

import { NoirCompilationArtifacts } from '../../noir_artifact.js';
import { NoirCompilationResult, NoirProgramCompilationArtifacts } from '../../noir_artifact.js';
import { NoirDependencyManager } from './dependencies/dependency-manager.js';
import { GithubDependencyResolver as GithubCodeArchiveDependencyResolver } from './dependencies/github-dependency-resolver.js';
import { LocalDependencyResolver } from './dependencies/local-dependency-resolver.js';
Expand Down Expand Up @@ -54,9 +54,6 @@ export class NoirWasmContractCompiler {
}

const noirPackage = NoirPackage.open(projectPath, fileManager);
if (noirPackage.getType() !== 'contract') {
throw new Error('This is not a contract project');
}

const dependencyManager = new NoirDependencyManager(
[
Expand All @@ -80,22 +77,73 @@ export class NoirWasmContractCompiler {
}

/**
* Compiles the project.
* Compile EntryPoint
*/
public async compile(): Promise<NoirCompilationArtifacts[]> {
const isContract = this.#package.getType() === 'contract';
// limit to contracts-only because the rest of the pipeline only supports processing contracts
if (!isContract) {
throw new Error('Noir project is not a contract');
public async compile(): Promise<NoirCompilationResult[]> {
if (this.#package.getType() === 'contract') {
this.#debugLog(`Compiling Contract at ${this.#package.getEntryPointPath()}`);
return await this.compileContract();
} else if (this.#package.getType() === 'bin') {
this.#debugLog(`Compiling Program at ${this.#package.getEntryPointPath()}`);
return await this.compileProgram();
} else {
this.#log(
`Compile skipped - only supports compiling "contract" and "bin" package types (${this.#package.getType()})`,
);
return [];
}
}

/**
* Compiles the Program.
*/
public async compileProgram(): Promise<NoirProgramCompilationArtifacts[]> {
await this.#dependencyManager.resolveDependencies();
this.#debugLog(`Dependencies: ${this.#dependencyManager.getPackageNames().join(', ')}`);

initializeResolver(this.#resolveFile);

try {
const isContract: boolean = false;
const result = compile(this.#package.getEntryPointPath(), isContract, {
/* eslint-disable camelcase */
root_dependencies: this.#dependencyManager.getEntrypointDependencies(),
library_dependencies: this.#dependencyManager.getLibraryDependencies(),
/* eslint-enable camelcase */
});

if (!('program' in result)) {
throw new Error('No program found in compilation result');
}

return [{ name: this.#package.getNoirPackageConfig().package.name, ...result }];
} catch (err) {
if (err instanceof Error && err.name === 'CompileError') {
this.#processCompileError(err as CompileError);
}

throw err;
}
}

/**
* Compiles the Contract.
*/
public async compileContract(): Promise<NoirCompilationResult[]> {
if (!(this.#package.getType() === 'contract' || this.#package.getType() === 'bin')) {
this.#log(
`Compile skipped - only supports compiling "contract" and "bin" package types (${this.#package.getType()})`,
);
return [];
}
this.#debugLog(`Compiling contract at ${this.#package.getEntryPointPath()}`);
await this.#dependencyManager.resolveDependencies();
this.#debugLog(`Dependencies: ${this.#dependencyManager.getPackageNames().join(', ')}`);

initializeResolver(this.#resolveFile);

try {
const isContract: boolean = true; // this.#package.getType() === 'contract';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT about removing the commented out code at the end? I like the explicit name for the parameter (here and in compileProgram), makes it clear what we're passing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops yeah will remove!

const result = compile(this.#package.getEntryPointPath(), isContract, {
/* eslint-disable camelcase */
root_dependencies: this.#dependencyManager.getEntrypointDependencies(),
Expand Down
9 changes: 9 additions & 0 deletions yarn-project/noir-compiler/src/compile/noir/package.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,13 @@ export class NoirPackage {
return this.#packagePath;
}

/**
* Gets this package's Nargo.toml (NoirPackage)Config.
*/
public getNoirPackageConfig() {
return this.#config;
}

/**
* The path to the source directory.
*/
Expand All @@ -44,6 +51,8 @@ export class NoirPackage {

switch (this.getType()) {
case 'lib':
// we shouldn't need to compile `lib` type, since the .nr source is read directly
// when the lib is used as a dependency elsewhere.
entrypoint = 'lib.nr';
break;
case 'contract':
Expand Down
Loading