Compiling in TS - refactor

[signorecello]

Problem

Intro

Currently, the recommended user flow for Noir is:

1. Write circuit
2. Use Nargo to test, prove, verify, generate contract, etc

Nargo is meant to be used as a CLI tool. Using it involves installing Nargo and running commands on an interactive shell. That was enough until a few months ago.

However, production apps need a more programmatic interface. For many reasons, we have concluded that JS is the programmatic support we will be giving, even though significant support is being given to other ways of using Noir (for example with NoirRS).

This comes with a problem: how to make Rust behave with JS? Turns out Rust is a pretty cool language, and tools like wasm-bindgen really make this easy. However, because JS is not a pretty cool language, most of the issues arise on this side of the equation. Let's lay them down:

Module system

Currently your NoirJS packages are written in TS and transpiled to both esm and cjs with tsc-multi.

As for the packages that use WASM, those are simply Rust that is compiled with wasm-bindgen to both targets node.js (cjs) and web(esm). Then some script runs to modify package.json to deal with the imports.

noir_wasm

This is one of the latter. It is rust that is compiled to a JS package on both cjs and esm. And more importantly, it needs to be WASM because that's how it interacts with the actual compiler that runs in Rust.

However, three big problems exist with WASM:

1. It can't access the internet. If my program requires another Noir package in a github repo, the wasm can't fetch it.
2. It requires instantiation. The most compatible way of using wasm-bindgen is the --target flag, which creates a .wasm file, then a .js "glue". The problem is that most bundlers like webpack or vite need to be told exactly how to work with non-js files. Even if all goes well, you still need to instantiate the wasm before actually calling functions... If you're cjs, because for esm you don't need to. Fucking. Mess.
3. It can't wait. It actually can wait, but using another tool that converts Promises to Futures. JS event-based architecture makes promises quite powerful, but apparently using this feature will introduce bugs in the compiler

source-resolver

To address this, we have source-resolver, a JS package that will resolve dependencies in JS, before calling noir_wasm. It fetches them on a filesystem, from the internet, etc. Then you pass the actual text of these files to noir_wasm, which does its thing. However, this is not so easy. Two examples:

1. Fetching things is a headache. Older versions of node don't have fetch. esm has import.meta.url but not __dirname. You can't use fs or path on the browser regardless of which module system you have. And many many many more problems.
2. You can't easily map dependencies. What if some dependencies are in filesystem, and other are in a github repo? What if main requires depA and depC, and depA requires depB and depC, how should you treat depC, as dependency of main, or as dependency of depA?
3. It needs to run synchronously. The reason is that because wasm can't wait, you can't rely on JS powerful event loop. If you need to fetch a dependency from the internet, you literally have to stop and wait for the call to come back, you can't just keep doing other things while you wait, like you usually do in JS. This blocks everything, you can't even scroll a page while it fetches, I shit you not.

Here's an example of how messy it quickly becomes:

// let's pretend I didn't spend 1h trying to make the module system work

async function getFileContent(path: string): Promise<string> {
  const file = readFileSync(path, "utf8")
  return file
}

export const compileCode = async () => {
  const [main, rsa, biguint, biguintUtils] = await Promise.all([
        getFileContent("./dkim/src/main.nr"),
        getFileContent("../crates/rsa/src/lib.nr"),
        getFileContent("../crates/rsa-biguint/src/lib.nr"),
        getFileContent("../crates/rsa-biguint/src/utils.nr"),
      ]);

      initializeResolver((file: string) => {
        if (file.endsWith("main.nr")) {
          return main;
        } else if (file.endsWith("/biguint/lib.nr")) {
          return biguint;
        } else if (file.endsWith("/biguint/utils.nr")) {
          return biguintUtils;
        } else if (file.endsWith("/rsa/lib.nr")) {
          return rsa;
        }
        return ""
      });

  const compiled = await compile(path.resolve("dkim/src", `main.nr`), false,{
    root_dependencies: ['biguint', 'rsa'],
    library_dependencies: {
      rsa: ['biguint'],
    },
  });

  return compiled;
};

I hope I have convinced you that this dev experience is probably the worst imaginable. Before even starting to use Noir, the user needs to check:

• If they're using TS
• Their module system
• Their environment (server or web)
• Their bundling needs
• What packages will they need and where are they stored

And then starting to think how exactly will they fit all this into their project. Because if they're bundling it or serving in a browser... Damn, RIP

Happy Case

Happy case is to have a good devex, abstracting away most of the stuff from the user:

• Correctly dealing with TS and esm and cjs in a way that user doesn't have to care which version are they using
• Abstracting away dependency resolution, so user just tells noir_wasm where's the project's Nargo.toml and forget about it
• Making it all compatible with the standard JS development flow

My naive suggestion would be to:

1. Make noir_wasm resolve dependencies in js, basically reading the Nargo.toml and recursively resolving dependencies, whether remote or local. Yes, this means eliminating source-resolver.
2. Getting to the bottom of why can't we correctly map Promises to Futures and be 10000% sure there's absolutely no other way to resolve the arising bugs
3. Taking care of wasm initialization in noir_wasm, and bundling that as a cross-module package

Alternatives Considered

No response

Additional Context

No response

Would you like to submit a PR for this Issue?

No

Support Needs

No response

[Savio-Sou]

Thank you for the great writeup!

Making it all compatible with the standard JS development flow

Is this simply referring to compatibility between the compiler + dependency resolver package coming out of this <> NoirJS and BarretenbergBackend?

Or is it referring to some general JS workflow?

[signorecello]

I mean with the way JS developers are used to write things, i.e. being aware of the event loop. To give a practical example, on the playground all I wanted is for a message to be displayed when the user hits the Compile button. As a JS developer, I would write it like:

// using toast
await toast.promise(compile(code), { pending: "Compiling...", success: "Compiled!" })

or

// using native "alert" function
alert("Compiling...")
await compile(code)
alert("Compiled!")

• User hits the button
• Message is displayed while the function waits for its result
• An event comes back when the function is resolved, displaying the success case

This is what I would define as "standard JS development flow".

With the current state of noir_wasm, instead, and because compile is a blocking call, it was actually deferring the alert for when the event loop stack was empty, engaging in the synchronous function compile that would completely block my page, and when that was done yes it would flush the stack and showing both alerts at the same time.

So I had to use some JS fuckery:

const compileTO = new Promise((resolve, reject) =>
  setTimeout(async () => {
    try {
      await compile(code);
      resolve(code);
    } catch (err) {
      reject(err);
    }
  }, 0)
);

await toast.promise(compileTO, {
  pending: "Compiling...",
  success: "Compiled!",
  error: "Error compiling",
});

In this case, I wrapped the synchronous call in a promise so it could be awaited, but because the toast is a promise itself, it would actually come later on as well. I had to double-wrap the compile call in a lower-prio construct such as a setTimeout with a ridiculous timeout of 0, just to make it flush sooner. And still it kept unnecessarily blocking the event loop, although it did trigger the alert beforehand

[kevaundray]

@signorecello can we close this?