WASM as a supported architecture

[wolfv]

I am opening an issue to track the state of wasm as a supported architecture. There have been a few discussions here and there about "what it would mean if conda-forge would support wasm" as a platform.

It would be cool if conda-forge had compiler activation scripts to cross-compile from linux-64 or other common architectures to wasm-32 (or should it be linux-wasm32?). The clang compiler can already do that.

Furthermore, we probably need a "standard libc" library. I think wasi-libc is such a candidate. Locally I have a recipe, it's really easy to compile.

The way I understand how stuff works is that we would add a new architecture and a "cross-compiler", so that e.g. build will be linux-64 and host will be wasm-32. Then the {{ compiler('c') }} will resolve to <compilername>_wasm-32 where compiler name will be either some default or chosen from the conda_build_config.yaml. In our case it should probably be clang. That compiler package would have a dependency on the real clang and a dependency (or contain) activation scripts for setting up CMake and autogen variables to do cross-compilation (those activation scripts would probably be derived from ctng-compiler-activation-feedstock, right? The compiler would also push a libc dependency into the host and run envs.

Once we have some successful cross-compiled packages, we can run them using tools like wasmer (or in the browser). However, wasmer would need to be installed with the host-architecture, so it's currently unclear how we would use this, ideally (stacking envs? one with linux-64, one with wasm-32?). But this discussion can be deferred right now.

There is also the alternative of using emscripten. As far as I understand, emscripten comes with some magic: a libc implementation, and it uses compiler-wrappers (as well as cmake and autotools wrappers). It will also generate some javascript entrypoints that make binaries runnable directly under node IIRC. However, I think that for conda-forge the "pure cross-compilation" with clang would be more desirable.

Any input / comments are appreciated! Would be cool to get something going :)

cc @bollwyvl @isuruf @SylvainCorlay @rth

[bollwyvl]

@jtpio and i have been playing with statically-hostable jupyterlab-like things, and having a solid build chain would be amazing. But that's just more "i want," not "here's how i can help!"

There's still a lot to do on wasmer (language bindings, "lite" build, etc), and we might even want to evaluate some of the other runtimes. If something would be helpful there, I'm all ears (and hopefully some fingers).

I really don't know much about bootstrapping an architecture... but that certainly seems like a better approach than having wasm-this and wasm-that. Maybe the wasmer (or whatever) runtime could be in a host-specific environment, and a wasm-32 environment would just be next to it? Or maybe we treat it differently entirely: a WASMforge distribution (still platform-specific) that created a base which contained just the runtime, but then only could install wasm packages? Sounds weird.

[rth]

There have been a few discussions here and there

Might be worth linking some of the previous discussions to avoid repeating them :)

Once we have some successful cross-compiled packages, we can run them using tools like wasmer (or in the browser).

I guess the question is what is the end goal. If the objective is to run with WASI on architectures that are currently not supported by conda-forge (of for sandboxing), then wasmer is likely the way to go. If it is to run in Node or browser, I wonder how ready it is and if anyone tried to build large packages with wasmer for that use case. There is https://github.com/wasmerio/wasmer-js but it doesn't look really active.

There is also the alternative of using emscripten [...] However, I think that for conda-forge the "pure cross-compilation" with clang would be more desirable.

+1. But it also depends what you want to do. For instance, you won't be able to run any Python compiled to WASM in the browser without the virtual filesystem integration provided by emscripten, or at least not without a lot more work.

[wolfv]

Ok, two links I managed to find:

• Adding a WebAssembly platform conda/conda#7619
• Better integration with conda/conda-forge for building packages pyodide/pyodide#795

Thanks for the pointer about emscripten filesystem. I need to read up a bit on how all of this works. The wasi-libc implementation of the POSIX filesystem doesn't do enough emulation to run in the browser?

I found this article quite interesting: https://depth-first.com/articles/2019/10/16/compiling-c-to-webassembly-and-running-it-without-emscripten/ It shows how to write some manual JS wrappers for wasi-libc code. I imagine that there could be a tool / we could produce a tool that writes these wrappers automatically, similar to how emscripten does it.