Better WebAssembly integration

[xtuc]

It seems that the current WebAssembly integration imposes some restriction: #647.

💬 RFC

In order to provide the best integration we want wasm to be part of the ES module graph, and treated as any other ES module. It has some issues at the moment, you can find more details here https://github.com/WebAssembly/esm-integration (that we can avoid with some transformations).

Parcel don't necessarily need to expose the importObject since it can be automatically wired for the user.

I propose you to use https://github.com/xtuc/webassemblyjs (disclaimer: i'm the author) to decode the wasm and apply transformations if needed (remove the start func, runtime, optimization etc).

🔦 Context

I'm currently working on the WebAssembly integration in Webpack.

💻 Examples

• wasm decoding and introspection: https://github.com/webpack/webpack/blob/master/lib/wasm/WebAssemblyParser.js

• transformation example: https://github.com/webpack/webpack/blob/master/lib/wasm/WebAssemblyGenerator.js

[19h]

I propose you to use https://github.com/xtuc/webassemblyjs (disclaimer: i'm the author) to decode the wasm and apply transformations if needed (remove the start func, runtime, optimization etc).

I'd argue against mutations in the bundling process. Toolchains like wasm-bindgen (Rust) already process WebAssembly and generate necessary bindings between Javascript and WebAssembly.

The biggest problem with bundling WebAssembly modules is the asynchronous nature of their compilation when they are inlined as base64 and exceed 4KB. It could be solved by restricting the flexibility of how those modules are loaded, reducing the complexity of building such an integration.

Given that you're working on the WebAssembly integration in Webpack, how did you approach it? And what did you settle on?

[xtuc]

I'd argue against mutations in the bundling process [...]

Yes, the less we mutate the module the less can go wrong, but it was required in Webpack to fit into the ESM module system.

Toolchains like wasm-bindgen (Rust) [...]

It's tight to Rust, I don't expect using it for bundling any time soon.

The biggest problem with bundling WebAssembly modules is the asynchronous nature of their compilation [...]

That's wrong, compilation can happen synchronously, even if not recommended (ok for service workers).

[...] when they are inlined

Inlining modules is not recommended because modules tends to be big.

[...] and exceed 4KB

Some engines have added a limitation for synchronous compilation. That's not an issue if we use instantiateStreaming.

[19h]

Spawning a Worker to execute something synchronously isn't exactly synchronous. instantiateStreaming is also asynchronous because it's returning a Promise.

(For Node.js specifically --) Parcel currently inlines fs.readFileSync as Buffer('<base64>') which is reasonable because it doesn't change the behaviour of it being a synchronous operation. But it breaks the WebAssembly instantiation if the module is bigger than 4KB. It can't magically rewrite the Buffer into being a ReadableStream that suddenly makes everything asynchronous -- or change the program code so that it suddenly includes an asynchronous WebAssembly loader. That brings me back to "restricting the flexibility of how those modules are loaded."

If you're using fetch to load the WebAssembly module there's no point in bundling it other than identifying the respective external resource and - if at all - rename the reference to carry a checksum.

Finally, if you need to mutate a WebAssembly module in order to fit a specific loader, and make that mutation default for the WebAssembly file type, then it would be better not to have it at all. Instead, this should be built as an extension for parcel that handles this specific use-case for users who choose to opt-in to it.

[xtuc]

According to

parcel/src/builtins/loaders/browser/wasm-loader.js

Line 5 in 9d6061c

return WebAssembly.instantiateStreaming(res);

and

parcel/src/builtins/loaders/node/wasm-loader.js

Line 14 in d272748

return WebAssembly.instantiate(data);

, none of them are inlining the module.

[19h]

That's precisely not what I highlighted.

Parcel currently inlines fs.readFileSync as Buffer('<base64>')

const join = require('path').join;
const bytes = require('fs').readFileSync('./foo.wasm');
const wasmModule = new WebAssembly.Module(bytes);
const wasmInstance = new WebAssembly.Instance(wasmModule, imports);
module.exports = wasmInstance.exports;

becomes

const join = require('path').join;
const bytes = Buffer("abcdef..");
const wasmModule = new WebAssembly.Module(bytes);
const wasmInstance = new WebAssembly.Instance(wasmModule, imports);
module.exports = wasmInstance.exports;

because Parcel generally inlines fs.readFileSync

[19h]

jfyi, this looks interesting: https://github.com/koute/parcel-plugin-cargo-web

[lygstate]

Is that possible to getting webassembly embbed as a string literal, not reading from filesystem? or fetch from remote URI.

[littledan]

Note, I've been working on the WebAssembly/ESM integration standard, and I'd like to bring alignment between Parcel and the future built-in browser behavior. You can find the proposal at https://github.com/WebAssembly/esm-integration/blob/master/proposals/esm-integration/README.md . I'm not sure if this is the same as Parcel semantics or something different. Any feedback you have would be welcome.

[littledan]

@lygstate You can use data URLs to embed WebAssembly as a string literal in the module specifier (but you might hit size limits). You can use CORS to fetch modules cross origin.

[lygstate]

@littledan I know this idea, but need parcel have a option support it

[devongovett]

I think we'd like to move forward with better WASM integration in Parcel 2. If someone would like to write up an RFC for what we'd need to do exactly and own this feature, that would be much appreciated!

[xtuc]

I would need to take a look at Parcel generated code, because we might need to rewrite the Wasm module a bit or add restrictions.

For context, when instantiating a Wasm module, the imported JavaScript values are copied/snapshoted, as opposed to take their references.
That means the JavaScript value must have been evaluated/initialized before the instantiation.

Similar to JavaScript Modules, if you rely on a linking phase and try to instantiate the module that won't work (what webpack does). Rollup on the other hand will inline/hoist the dependecies before the instantions which just works.

[samvv]

Given that I don't have a project to work on aside from my own, I'm willing to give it a try. I'll start by implementing a naive approach to see where the issues lie and then come back with an RFC.

[mischnic]

I think the most flexible and least non-standard integration would be something like

• import {a,b,c} from "./x.wasm" = the WASM ESM spec
• import {url, instantiate} from "parcel-wasm:./x.wasm" = to set memory/imports for more advanced usecases

[Yongle-Fu]

lgarron/parcel-wasm-window-load-bug#1

[Yongle-Fu]

No transformers found for add.wasm.

[chinoto]

@mischnic The way the plugin for Vite works is rather nice:

import init, { Universe, Cell } from "wasm-game-of-life";
let { memory } = await init();

^ Example from following the WASM Game of Life guide.