SUPPORT_BIG_ENDIAN=1 causes internal compiler error ReferenceError: maybeExport is not defined

[merceyz]

Starting with version 3.1.55 compiling with -s SUPPORT_BIG_ENDIAN=1 fails with ReferenceError: maybeExport is not defined.

Version of emscripten/emsdk:

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.57 (1df9c1977b49926c1efca672c31414da45c0c7bb)
clang version 19.0.0git (https:/github.com/llvm/llvm-project ccdebbae4d77d3efc236af92c22941de5d437e01)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /emsdk/upstream/bin

Failing command line in full:

touch test.c && docker run --rm -it -v "$(pwd):/src" emscripten/emsdk:3.1.57 emcc \
  -o ./build.js \
  -s SUPPORT_BIG_ENDIAN=1 \
  ./test.c

Full link command and output with -v appended:

touch test.c && docker run --rm -it -v "$(pwd):/src" emscripten/emsdk:3.1.57 emcc \
  -o ./build.js \
  -s SUPPORT_BIG_ENDIAN=1 \
  -v \
  ./test.c

 /emsdk/upstream/bin/clang -target wasm32-unknown-emscripten -fignore-exceptions -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --sysroot=/emsdk/upstream/emscripten/cache/sysroot -DEMSCRIPTEN -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -v ./test.c -c -o /tmp/emscripten_temp_9iwde_5q/test_0.o
clang version 19.0.0git (https:/github.com/llvm/llvm-project ccdebbae4d77d3efc236af92c22941de5d437e01)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /emsdk/upstream/bin
 (in-process)
 "/emsdk/upstream/bin/clang-19" -cc1 -triple wasm32-unknown-emscripten -emit-obj -mrelax-all -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name test.c -mrelocation-model static -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-cpu generic -debugger-tuning=gdb -fdebug-compilation-dir=/src -v -fcoverage-compilation-dir=/src -resource-dir /emsdk/upstream/lib/clang/19 -D EMSCRIPTEN -isysroot /emsdk/upstream/emscripten/cache/sysroot -internal-isystem /emsdk/upstream/lib/clang/19/include -internal-isystem /emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten -internal-isystem /emsdk/upstream/emscripten/cache/sysroot/include -Werror=implicit-function-declaration -ferror-limit 19 -fvisibility=default -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fignore-exceptions -fcolor-diagnostics -iwithsysroot/include/fakesdl -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o /tmp/emscripten_temp_9iwde_5q/test_0.o -x c ./test.c
clang -cc1 version 19.0.0git based upon LLVM 19.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten"
#include "..." search starts here:
#include <...> search starts here:
 /emsdk/upstream/emscripten/cache/sysroot/include/fakesdl
 /emsdk/upstream/emscripten/cache/sysroot/include/compat
 /emsdk/upstream/lib/clang/19/include
 /emsdk/upstream/emscripten/cache/sysroot/include
End of search list.
 /emsdk/upstream/bin/clang --version
cache:INFO: generating system asset: symbol_lists/18d7bfe144a9d992041e0f5e344fef6ada2c0532.json... (this will be cached in "/emsdk/upstream/emscripten/cache/symbol_lists/18d7bfe144a9d992041e0f5e344fef6ada2c0532.json" for subsequent builds)
 /emsdk/node/16.20.0_64bit/bin/node /emsdk/upstream/emscripten/src/compiler.mjs /tmp/tmprmx6ym59.json --symbols-only
cache:INFO:  - ok
 /emsdk/upstream/bin/wasm-ld -o ./build.wasm -L/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten /tmp/emscripten_temp_9iwde_5q/test_0.o -lGL-getprocaddr -lal -lhtml5 -lstubs-debug -lnoexit -lc-debug -ldlmalloc -lcompiler_rt -lc++-noexcept -lc++abi-debug-noexcept -lsockets -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr /tmp/tmpsasqvevflibemscripten_js_symbols.so --strip-debug --export=emscripten_stack_get_end --export=emscripten_stack_get_free --export=emscripten_stack_get_base --export=emscripten_stack_get_current --export=emscripten_stack_init --export=_emscripten_stack_alloc --export=__get_temp_ret --export=__set_temp_ret --export=__wasm_call_ctors --export=_emscripten_stack_restore --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=main --export-if-defined=__main_argc_argv --export-if-defined=fflush --export-table -z stack-size=65536 --no-growable-memory --initial-heap=16777216 --no-entry --stack-first --table-base=1
 /emsdk/upstream/bin/llvm-objcopy ./build.wasm ./build.wasm --remove-section=.debug* --remove-section=producers
 /emsdk/upstream/bin/wasm-emscripten-finalize --dyncalls-i64 --pass-arg=legalize-js-interface-exported-helpers ./build.wasm -o ./build.wasm --detect-features
 /emsdk/node/16.20.0_64bit/bin/node /emsdk/upstream/emscripten/src/compiler.mjs /tmp/tmpqmhbwdg8.json
Internal compiler error in src/compiler.js!
Please create a bug report at https://github.com/emscripten-core/emscripten/issues/
with a log of the build and the input files used to run. Exception message: "preamble.js:1
 maybeExport('HEAP_DATA_VIEW') 
 ^

ReferenceError: maybeExport is not defined
    at preamble.js:1:2
    at Script.runInContext (node:vm:141:12)
    at Script.runInNewContext (node:vm:146:17)
    at Module.runInNewContext (node:vm:306:38)
    at runInMacroContext (file:///emsdk/upstream/emscripten/src/utility.mjs:331:13)
    at file:///emsdk/upstream/emscripten/src/parseTools.mjs:37:17
    at String.replace (<anonymous>)
    at processMacros (file:///emsdk/upstream/emscripten/src/parseTools.mjs:36:15)
    at getIncludeFile (file:///emsdk/upstream/emscripten/src/jsifier.mjs:144:15)
    at includeFile (file:///emsdk/upstream/emscripten/src/jsifier.mjs:683:11)
emcc: error: '/emsdk/node/16.20.0_64bit/bin/node /emsdk/upstream/emscripten/src/compiler.mjs /tmp/tmpqmhbwdg8.json' failed (returned 1)

[sbc100]

It looks like that function was renamed in #21439 but that call was overlooked. I guess we don't have any testing for that setting :-/

Can I ask what your target is that you are using that option?

[merceyz]

Can I ask what your target is that you are using that option?

Of course!

Yarn compiles libzip to WASM for Node.js and needs to support big-endian systems.
I don't know specifically what systems people run it on but I know at least big-endian s390x systems are in use.

We don't have access to a big-endian system so our CI emulates one while Node.js CITGM validates that it works on a real machine.

[sbc100]

Thanks! Thats good to know. I didn't know yarn was using wasm like this. BTW, can I ask why you/they didn't just go with a native node binary? What is the advantage they/you seen in using wasm under node? (I'm not saying there are not advantages, I'm just wondering why you made the choice).

[merceyz]

The main reason is reproducibility. We need the output of libzip to be identical on all systems since we checksum the zip files to see if they've been tampered with or corrupted.

It's also easier to compile to WASM once and use it everywhere Node.js works than to make a native Node.js addon for each and every architecture, platform, and libc implementation that Node.js runs on.

[sbc100]

I see, I'm not very familiar with node addon.

Does that mean that npm doesn't generally do compile on install for things like? Does it instead rely on your to compile native extension for evey possible arch up front? That does sounds hard/impossible.

Also, just to confirm, the SUPPORT_BIG_ENDIAN setting means that result will run on both big and little endian with a single build?

I guess we should add more testing of this setting if folks like you are depending on it so much. Hopefully it doesn't come with too much of a performance hit.

[merceyz]

Compile on install can be done but then you risk the host system not having the required tools or correct version of tools to build the library.

Some npm packages only ship pre-compiled versions and fail if it doesn't have one for the host and Node.js version, while some fall back to compile on install.

It's easier for us to ignore all that and compile once to WASM.
It's also faster for our users since they don't have to compile anything when installing Yarn.

Also, just to confirm, the SUPPORT_BIG_ENDIAN setting means that result will run on both big and little endian with a single build?

Yes.

Hopefully it doesn't come with too much of a performance hit.

When support for big-endian was added to Yarn the benchmarks indicated almost no impact on performance yarnpkg/berry#3669 (comment).

[sbc100]

Thanks for the useful data points!