Wasm es6 modules fail to load with Jest, but work in Node #11011

modulitos · 2021-01-16T20:00:52Z

🐛 Bug Report

I'm able to load an es6 module with Node, but I'm unable to load the same module when running with Jest. When I try loading the module with Jest, I get the following error:

node --experimental-modules --experimental-wasm-modules --experimental-vm-modules node_modules/.bin/jest

(node:72954) ExperimentalWarning: VM Modules is an experimental feature. This feature could change at any time
(Use node --trace-warnings ... to show where the warning was created)
FAIL tests/test_jest.test.mjs
● Test suite failed to run
/Users/me/projects/bin_packing/jest-wasm/node_modules/bin_packer_3d/bin_packer_3d_wasm_bg.wasm:1
({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){


SyntaxError: Invalid or unexpected token

  at Runtime.createScriptFromCode (node_modules/jest-runtime/build/index.js:1350:14)

To Reproduce

Steps to reproduce the behavior:

Clone this repo: https://github.com/modulitos/jest-wasm-esm-example
run npm run test-jest

Note that running npm run test-node loads the module successfully in Node. So why does it fail with Jest?

Expected behavior

The module loads when running npm run test-jest.

Link to repl or repo (highly encouraged)

https://github.com/modulitos/jest-wasm-esm-example

envinfo

❯ npx envinfo --preset jest
npx: installed 1 in 1.358s

System:
OS: macOS 10.15.7
CPU: (4) x64 Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
Binaries:
Node: 14.15.4 - ~/.asdf/installs/nodejs/14.15.4/bin/node
npm: 6.14.10 - ~/.asdf/installs/nodejs/14.15.4/bin/npm
npmPackages:
jest: ^26.6.3 => 26.6.3

The text was updated successfully, but these errors were encountered:

ahnpnl · 2021-01-18T14:18:24Z

The error is about Jest module resolution tries to resolve bin_packer_3d as bin_packer_3d_wasm_bg.wasm while NodeJS module resolution resolves as bin_packer_3d_wasm_bg.js. Adjust moduleNameMapper to map bin_packer_3d to bin_packer_3d_wasm_bg.js should fix the error.

However, after correcting Jest module resolution, I ran into another error

 RUNS  __tests__/test_jest.test.mjs
Exception in PromiseRejectCallback:
/Users/ahn/jest-wasm-esm-example/node_modules/jest-runtime/build/index.js:586
    return module;

RangeError: Maximum call stack size exceeded
    at loadEsmModule (/Users/ahn/jest-wasm-esm-example/node_modules/jest-runtime/build/index.js:586:19)
    at linkModules (/Users/ahn/jest-wasm-esm-example/node_modules/jest-runtime/build/index.js:616:19)
    at /Users/ahn/jest-wasm-esm-example/node_modules/jest-runtime/build/index.js:572:18
    at node:internal/vm/module:320:30
    at SourceTextModule.<computed> (node:internal/vm/module:319:36)
    at link (node:internal/vm/module:197:22)
    at loadEsmModule (/Users/ahn/jest-wasm-esm-example/node_modules/jest-runtime/build/index.js:571:12)
    at linkModules (/Users/ahn/jest-wasm-esm-example/node_modules/jest-runtime/build/index.js:616:19)
    at /Users/ahn/jest-wasm-esm-example/node_modules/jest-runtime/build/index.js:572:18

which Idk how to solve.

If simply solving your original error, moduleNameMapper is enough.

modulitos · 2021-02-06T07:48:57Z

Thank you for the tip! I updated the repo with the correct moduleNameMapper config, and can confirm that I'm getting the same RangeError: Maximum call stack size exceeded problem.

Any suggestions for how I can troubleshoot this issue is appreciated 🙏

ahnpnl · 2021-02-06T12:37:08Z

Just now I noticed that it’s a combination of wasm + ESM. Maybe you should try jest@next first and if not working then go to the ESM tracking issue to mention this issue there.

modulitos · 2021-02-10T07:27:52Z

Ah, thanks again for the insight. Can you link me to the "ESM tracking issue"? I'm not sure what you're referring to, but happy to mention this issue there.

I updated to jest@next (updated the repo linked above as well), but now I'm getting this error:

TypeError: wasm.binpacker_packing_algorithm is not a function

It's a different error, but it seems like we're getting closer! But I'm still not sure how to debug this 🤔

ahnpnl · 2021-02-10T07:44:39Z

#9430

SimenB · 2021-02-10T08:12:37Z

Jest does not support WASM at this time. I'm not sure how support would be added. Is it enough to do WebAssembly.compile on the source code (sorta like the example here)?

I have zero idea how WASM works, so I can't really do much here. Are there any JS globals available in WASM? If not I guess just executing the file and returning the result should be enough?

SimenB · 2021-02-10T08:24:09Z

I guess reading https://github.com/WebAssembly/esm-integration/tree/master/proposals/esm-integration is a start

(I can almost guarantee I won't have the time or energy to work on this for some time, so if anybody wants to pick it up that'd be great!)

modulitos · 2021-02-12T08:31:07Z

Thanks everyone for the clarification! This makes much more sense now.

I didn't realize that WASM isn't supported in Jest - I was a little naive in thinking that if I can load a module in Node, then I should be able to load it in Jest.

I appreciate the help 🙏

SimenB · 2021-02-12T15:50:54Z

I was a little naive in thinking that if I can load a module in Node, then I should be able to load it in Jest.

That's how it's supposed to work, but node has added a bunch of stuff for ESM (including WASM) in the last couple of years we haven't been able to follow (especially since most is still behind flags on the Node side)

FlatAssembler · 2021-06-22T11:47:33Z

Are there any JS globals available in WASM?

No, JavaScript global variables are not accessible from WebAssembly by default. The earliest versions of WebAssembly did not have global variables at all. Modern WebAssembly does have global variables, and they can be shared between JavaScript and WebAssembly.

gthb · 2021-10-30T11:20:44Z

Is anyone currently working on this?

knightcode · 2022-05-24T16:48:27Z

How have our tests been working with the WASM in virgil-crypto for months and just now started to fail?

kachkaev · 2022-10-13T19:54:44Z

I managed to tweak jest-runtime and this allowed me to import *.wasm from native ESM modules 🎉

Demo repo: hasharchives/wasm-ts-esm-in-node-jest-and-nextjs
The patch: jest-runtime-npm-29.1.2-567b60ffd5.patch

This is obviously not a production-grade solution, but I hope it can helpful to someone.

SimenB · 2022-10-13T20:00:04Z

@kachkaev happy to take a PR if you can add some tests 🙂 some support is way better than no support

kachkaev · 2022-10-13T20:10:10Z

My knowledge of WebAssembly and SyntheticModule APIs is extremely limited, so I’m not sure if my solution is good enough even for an MVP. It’d be great if someone a bit more experienced could sense-check the patch first.

My _importWasmModule function maps WebAssembly.Module imports and exports to an instance of SyntheticModule quite naively, based on just a couple of wasm file examples I’ve seen. It’s possible that I’m not covering some common cases, but I can’t tell that because I don’t know where to find more wasm files 😅

SimenB · 2022-10-13T20:20:47Z

Fair enough. It looks like you pretty much just mapped https://github.com/nodejs/node/blob/7cdf745fddade8b7fc644f44121fbba041d3ac4c/lib/internal/modules/esm/translators.js#L322-L349 to SyntheticModule, so seems fine to me. Let's try and if people hit cases that don't work adjust?

kachkaev · 2022-10-13T20:45:01Z

Interesting! I did not see this translator so was crafting the patch in the ‘blind watchmaker’ mode 😅

Not sure I’ll have capacity to create a PR in my work hours, so unfortunately I can start it only around 22-23 Oct (the earliest). I’d be happy if someone else makes an attempt before me – feel free to recycle the demo repo.

kachkaev · 2022-10-24T23:37:48Z

@SimenB I’ve started #13505, which essentially copies my experimental patch over to jest-runtime. I have to wrap up for today, but happy to continue the PR later this week if things go as planned. Any feedback welcome!

SimenB · 2022-11-07T18:00:03Z

https://github.com/facebook/jest/releases/tag/v29.3.0

kachkaev · 2022-11-07T20:12:03Z

Nice! Just tested 29.3.0 against the same demo project and it worked without a local patch! 🎉

knightcode · 2022-11-09T19:47:27Z

I'm still having a failure with virgil-crypto on node 16, which needs access to a hardware randomness (I assume... it complains about an "entropy source" failing). I'm not sure how it goes about accessing it, but it works when jest isn't involved.

kachkaev · 2022-11-09T23:12:19Z

There is a bug in the initial implementation, to do with imports into the Wasm module. Details here: #13505 (comment)

If you see

TypeError: modulePath.endsWith is not a function
  at isWasm (../../../node_modules/jest-runtime/build/index.js:210:41)

you can use this patch for now:

diff --git a/node_modules/jest-runtime/build/index.js b/node_modules/jest-runtime/build/index.js
index 9906023..d8458dd 100644
--- a/node_modules/jest-runtime/build/index.js
+++ b/node_modules/jest-runtime/build/index.js
@@ -1509,7 +1509,7 @@ class Runtime {
     const moduleLookup = {};
     for (const {module} of imports) {
       if (moduleLookup[module] === undefined) {
-        moduleLookup[module] = await this.loadEsmModule(
+        moduleLookup[module] = await this.linkAndEvaluateModule(
           await this.resolveModule(module, identifier, context)
         );
       }

UPD: Fixed via #13608

SimenB · 2022-11-10T09:14:10Z

@knightcode can you open up a new issue?

github-actions · 2022-12-13T00:08:31Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

modulitos added Bug Report Needs Repro Needs Triage labels Jan 16, 2021

SimenB added 🚀 Feature Request and removed Bug Report Needs Repro Needs Triage labels Feb 10, 2021

gthb mentioned this issue Sep 5, 2022

Jest failures rustwasm/wasm-bindgen#3066

Open

kachkaev mentioned this issue Oct 24, 2022

feat: support WebAssembly (Wasm) imports in ESM modules #13505

Merged

3 tasks

SimenB closed this as completed in #13505 Nov 6, 2022

kachkaev mentioned this issue Nov 12, 2022

Support Wasm files that import JS resources #13608

Merged

github-actions bot locked as resolved and limited conversation to collaborators Dec 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wasm es6 modules fail to load with Jest, but work in Node #11011

Wasm es6 modules fail to load with Jest, but work in Node #11011

modulitos commented Jan 16, 2021

ahnpnl commented Jan 18, 2021

modulitos commented Feb 6, 2021

ahnpnl commented Feb 6, 2021

modulitos commented Feb 10, 2021

ahnpnl commented Feb 10, 2021

SimenB commented Feb 10, 2021

SimenB commented Feb 10, 2021 •

edited

Loading

modulitos commented Feb 12, 2021

SimenB commented Feb 12, 2021

FlatAssembler commented Jun 22, 2021

gthb commented Oct 30, 2021

knightcode commented May 24, 2022

kachkaev commented Oct 13, 2022 •

edited

Loading

SimenB commented Oct 13, 2022

kachkaev commented Oct 13, 2022 •

edited

Loading

SimenB commented Oct 13, 2022 •

edited

Loading

kachkaev commented Oct 13, 2022 •

edited

Loading

kachkaev commented Oct 24, 2022 •

edited

Loading

SimenB commented Nov 7, 2022

kachkaev commented Nov 7, 2022

knightcode commented Nov 9, 2022

kachkaev commented Nov 9, 2022 •

edited

Loading

SimenB commented Nov 10, 2022

github-actions bot commented Dec 13, 2022

Wasm es6 modules fail to load with Jest, but work in Node #11011

Wasm es6 modules fail to load with Jest, but work in Node #11011

Comments

modulitos commented Jan 16, 2021

🐛 Bug Report

To Reproduce

Expected behavior

Link to repl or repo (highly encouraged)

envinfo

ahnpnl commented Jan 18, 2021

modulitos commented Feb 6, 2021

ahnpnl commented Feb 6, 2021

modulitos commented Feb 10, 2021

ahnpnl commented Feb 10, 2021

SimenB commented Feb 10, 2021

SimenB commented Feb 10, 2021 • edited Loading

modulitos commented Feb 12, 2021

SimenB commented Feb 12, 2021

FlatAssembler commented Jun 22, 2021

gthb commented Oct 30, 2021

knightcode commented May 24, 2022

kachkaev commented Oct 13, 2022 • edited Loading

SimenB commented Oct 13, 2022

kachkaev commented Oct 13, 2022 • edited Loading

SimenB commented Oct 13, 2022 • edited Loading

kachkaev commented Oct 13, 2022 • edited Loading

kachkaev commented Oct 24, 2022 • edited Loading

SimenB commented Nov 7, 2022

kachkaev commented Nov 7, 2022

knightcode commented Nov 9, 2022

kachkaev commented Nov 9, 2022 • edited Loading

SimenB commented Nov 10, 2022

github-actions bot commented Dec 13, 2022

SimenB commented Feb 10, 2021 •

edited

Loading

kachkaev commented Oct 13, 2022 •

edited

Loading

kachkaev commented Oct 13, 2022 •

edited

Loading

SimenB commented Oct 13, 2022 •

edited

Loading

kachkaev commented Oct 13, 2022 •

edited

Loading

kachkaev commented Oct 24, 2022 •

edited

Loading

kachkaev commented Nov 9, 2022 •

edited

Loading