Feature request: Support for wasm?

[NobodyXu]

I wonder is it possible for cargo-binstall to run on wasm target and add wasm target as a fallback if the environment supports running wasm?

[NobodyXu]

After a bit research, I think the target we want to support is wasi, which provides a std library.

Tokio is working on supporting it tokio-rs/tokio#4827 , once it's done we can port cargo-binstall to wasi.

Meanwhile we can add support for detecting wasi runtime by checking whether wasmtime or wasmer is present on the system.

[passcod]

Note that recent Node.js also supports wasi

[NobodyXu]

Another runtime we should support is WasmEdge.

In additional to existing wasi support, it also supports thread proposal, wasi-sockets and wasi crypto.

[NobodyXu]

@passcod Do you think it is necessary to have a target for wasi-wasmedge (or wasi-nodejs etc) in additional to normal wasi target since it has support for thread, sockets and crypto, which is just enough to run a reqwest::Client or a https server?

I know that this is not ideal and probably should be shoot down because wasm is supposed to be portable, but given its current status, perhaps having an target would make sense for now?

I also wonder whether it is a good idea to also support the following wasm target:

• wasm32-unknown-unknown this one is pretty portable, can access std library, but many I/O functions such as std::fs and std::net would simply return error.
  I think this can only be used for very limited functionalities so probably not.
• wasm64-unknown-unknown same as wasm32-unknown-unknown except only supported by nodejs and wasmtime, may have better performance but still very limited.
• wasm32-unknown-emscripten: This one supports std library, including TCP sockets/file io/multithreading/opengl so I think we probably should support it even it is considered "hacky" because it is the only target that support sockets, files and multithreading.

[passcod]

This is an area where it would be beneficial to find someone involved or familiar with the space for advice, I think.

[NobodyXu]

This is an area where it would be beneficial to find someone involved or familiar with the space for advice, I think.

I would investigate into this area first by trying to create wasm32-unknown-unknown and wasm32-unknown-emscripten.

I also discovered another wasm runtime WAVM that compiles wasm using llvm to native code.

According to emscripten's official doc, wasmtime, wasmer and WAVM all support wasm32-unknown-emscripten.

wasmtime explicitly does not support emscripten

[NobodyXu]

@passcod After a few investigation, I am fairly confident to say that wasm32-unknown-emscripten won't help us here.
It can be compiled to standalone mode, but that will be equivalent to wagi.
TO support net/multithreading, it must be run on browser.

So the questions left are:

• Do we need to support wasm32-unknown-unknown (and perhaps wasm64-unknown-unknown)?
• Do we need to have a target wasi-wasmedge in additional to normal wasi target since it has support for thread, sockets and crypto, which is just enough to run a reqwest::Client or a https server?

[NobodyXu]

After reading wasmtime's doc for rust, I think it is pretty clear that we only need to support wasi, not wasm32-unknown-unknown since even println! cannot be used on it.

[NobodyXu]

targets

I think we will want to support:

• wasm32-wasi
• wasm32-wasi-wasmedge
• wasm32-wasi-wasmtime
• wasm32-wasi-wavm

We will only attempt wasm32-wasi if no other target support the crate.
We would also prefer wasm32-wasi to wasm32-wasi-wasiedge because they are non-std.

The reason why wasmedge, wasmtime and wavm all have their special targets is because they support extensions that are essential but not yet standardized.

For example, wasmedge supports:

• wasmedge_process_interface
• wasmedge_wasi_socket
• wasmedge_tensorflow_interface

Same for wasmtime:

• wasmtime-wasi-nn
• wasi-experimental-http-wasmtime
• wasmtime-wasi-crypto

All of the extensions above are available in rust.

WAVM also supports extensions such as threading.

Detection

For wasmtime, wasmer, wasmedge and wavm, we would simply run $bin --version or something like that to detect their existence.

For nodejs, we will use node --experimental-wasi-unstable-preview1 -e "require.resolve('wasi')" to detect its wasi support.

New registry

After a bit googling, I found a wapm, a registry for wasm apps and libraries that can be run on wasmer.
It also has a GraphQL API which we can access using graphql_client.

[NobodyXu]

@passcod @ryankurte How do you think of my plan of supporting wasi?

[passcod]

Hmm, supporting a completely third party registry that's not crates is a big scope creep, especially since wapm already has its own tooling. I'd rather keep binstall focused on the Rust ecosystem for now; we can support installing crates that have wasi binaries built, but I think widening the scope to a completely different ecosystem is a jump too far.

Treating wapm as a third party repository of built wasi artifacts is fine, though... e.g. taking yozuk as example, they would add:

# zuk's Cargo.toml

[package.metadata.binstall.overrides.wasm32-wasi]
pkg-url = "https://registry-cdn.wapm.io/packages/yozuk/zuk/zuk-{ version }.{ archive-format }"
bin-dir = "target/wasm32-wasi/release/{ bin }.wasm"
pkg-fmt = "tgz"

Then if binstall detects that the host supports wasi, it could install that... somewhere. This raises the question of how we install these things, too: they're not executables per se, so they'd need some kind of wrapper? Is that our role, or should we instead trust the user is ok to set themselves up?

If we don't make wrappers, I don't think we can reasonably default to installing wasi even as a fallback at this juncture. Making wrappers raises question like what if the user uninstalls their wasi runtime or changes to another one?

Interestingly, with the above metadata added to zuk's Cargo.toml, it already works (if a bit awkward because of the lack of wrapper) with --target wasm32-wasi:

(I had to use the Cargo.toml for version 0.20.6 because 0.20.10 isn't published on wapm).

Also not really convinced by making up "virtual" targets. The main problem is these are moving targets: wasmedge etc may add extensions, they may also remove them as things get specified or stabilised. Further, there's a confusion between WASI "extensions" (really just APIs exposed by the host runtime) and WASM extensions (new features of the bytecode).

Thinking about it like native targets, it's similar to if a crate says "only works on x86 with the AVX-512 CPU extension". Binstall isn't concerned with figuring out either the CPU feature support of the host nor the CPU feature requirements of the crate. Some extensions are present in the target, like hf on ARM and gc on RISCV64, so it's not unprecedented, but if we went that way I'd rather make "virtual" targets with the actual features, not with vendor names... but ultimately this would have to be informed by what upstreams are doing.

So imo the right approach would be to go with a single target, wasm32-wasi, figure out some way of making wrappers that invoke the wasi runtime (maybe there's a standard already?), let upstreams declare where their wasi binaries are the normal way (afaics there's no way in wapm's graphql to query for a particular source repo), and once we've got that going we can inform future plans from how the feature is used.

[NobodyXu]

@passcod I agree that we should not worry about wapm registry or runtime specific extension and just support wasm32-wasi.

figure out some way of making wrappers that invoke the wasi runtime

On Linux, we can use binfmt_misc to register wasm interpreter to the kernel so that it will be automatically invoked when executing wasm.

Perhaps we should have a configuration file and let the user opt in to use wasm as a fallback?

[passcod]

Can we detect that wasi is executable somehow? That would provide the strongest hint that we can use wasi as fallback

Maybe we could embed the smallest valid wasi program, write it to a temp file, and try to run it, and if it does, we know we're good to go.

[NobodyXu]

That sounds plausible, though I'm not sure about whether there is anything similar to binfmt_misc on windows, mac or any oyher unix OS.

[NobodyXu]

But again, having wasi support on Linux is better than nothing and an opportunity to receive more feedback.

[NobodyXu]

Maybe we could embed the smallest valid wasi program, write it to a temp file, and try to run it, and if it does, we know we're good to go.

I think we can create a hello-world program, strip its symbols, disable backtrace and build it with opt-level z to get a binary as small as possible yet can detect support of wasi.

[passcod]

I think I can get one to under 200 bytes, working on it

[passcod]

Here we go. 97 bytes compiled:

(module
 (import "wasi_snapshot_preview1" "proc_exit" (func $exit (param i32)))
 (memory $0 16)
 (export "memory" (memory $0))
 (export "_start" (func $0))
 (func $0
  (call $exit (i32.const 0))
  (unreachable)
 )
)

[NobodyXu]

@passcod Can it tell wasm32-unknown-unknown from wasm32-wasi ?

[npmccallum]

cc @rjzak

[passcod]

@NobodyXu Not really a thing at this point because I'm writing WAST directly, but see my edit and the post where I add a wasi import

[NobodyXu]

@passcod This is absolutely amazing.
Only 97 bytes for a wasi program.

[passcod]

Re a wrapper, to widen support, the following idea comes from discussion in a discord I'm in:

• make a small binary that finds an available wasmtime or wasmer or other supported wasm runtime (or looks into a WASM_RUNTIME environment variable)
• said binary is prepended to the wasm program, essentially cat wrapper program > wrapped
• when run it finds its own file, its own end, and then the \0asm magic
• it makes a pipe:
  • on windows it creates a named pipe
  • on posix it uses mkfifo
• it invokes wasmruntime /path/to/pipe as a child process with std streams inherited
• it pipes the wasm blob into the pipe, then relays any signals and exit codes across

[NobodyXu]

This sounds good.

Also, here's a list of wasm runtime I found:

• wasmtime
• wasmer
• node.js
• wasmedge
• wavm
• graalvm
• enarx

[NobodyXu]

@passcod There's actually another way to support it, is to compile wasm to native asm.
I think wasmtime, wasmedge, wavm and graalvm support this.

[NobodyXu]

After investigation, wasmer also supports this.
Graalvm didn't mention whether it can compile wasm to native code and its wasm seems to also be experimental.

Perhaps we can ignore node.js and graalvm for now, since their wasm support is experimental, and then @passcod we can compile the wasi code into native code using wasmtime, wasmer, wasmedge or wavm.

[passcod]

For wasmtime and wasmer at least, I haven't managed to run the resulting ELF directly, you still need to use e.g. wasmtime run --allow-precompiled file.cwasm, and the compiled file isn't portable:

--allow-precompiled                                                                     
    Allow executing precompiled WebAssembly modules as `*.cwasm` files.                 
                                                                                        
    Note that this option is not safe to pass if the module being passed in is arbitrary
    user input. Only `wasmtime`-precompiled modules generated via the `wasmtime compile`
    command or equivalent should be passed as an argument with this option specified.   

Seems AOT compilation is to make runs faster but not necessarily to make a native binary?

[NobodyXu]

Seems AOT compilation is to make runs faster but not necessarily to make a native binary?

Oh I didn't try them, but this is a pity.
If they generate an actual runnable ELF, then we will be free of the wasm dependency and supporting it will be much easier.

BTW, does they compile it to a dynamic library *.so and load it?
And what about WAVM?

[passcod]

for these two at least they have their own format. though wasmer can generate .o or .so files too. haven't tried others yet

[NobodyXu]

for these two at least they have their own format. though wasmer can generate .o or .so files too. haven't tried others yet

It's the same on mac, wasmer on mac can generate .o or .dylib or the default format, which is not even elf.

[npmccallum]

@NobodyXu Don't forget enarx!

[passcod]

@npmccallum Do you have any feedback on the test program I have above? While in works in runtimes I have tested so far, it fails with enarx:

$ ./enarx-x86_64-unknown-linux-musl --version
enarx 0.6.1
$ ./enarx-x86_64-unknown-linux-musl run src/miniwasi.wasm
Error: Exited with i32 exit status 0
wasm backtrace:
    0:   0x5d - <unknown>!<wasm function 1>

Edit: Or, rather, I guess "exiting with status 0" is exactly what the program does, but I don't understand why that's considered an error to enarx

[npmccallum]

@passcod That's a good question. I have filed this bug: enarx/enarx#2094

Note (unrelatedly), however, that wat2wasm complains about your .wat file.

$ wat2wasm exit.wat
exit.wat:5:3: error: imports must occur before all non-import definitions
 (import "wasi_snapshot_preview1" "proc_exit" (func $exit (param i32)))
  ^^^^^^

[npmccallum]

@passcod We have fixed the above issue.

[NobodyXu]

@passcod Actually, let's create issues on wasmtime/wasmer to see if they are willing to provide functionalities to compile wasm to actually runnable executables, instead of an ELF lib.

[NobodyXu]

@passcod I've submit my first feature request to wasmtime bytecodealliance/wasmtime#4563

BTW, @npmccallum is the same thiing possible with enarx?
Or is there any other way to use the wasm binaries just like any native executables?

[passcod]

Their response is about what I was expecting, and I don't really see the point of pursuing that tbh. The cool thing about wasm is it's portable, and compiling it down to a less portable version is not really the point. The compiled+dylib versions, which would be the only reasonable way to do this, would likely require tight coupling to one particular runtime and version, or to define a single dynamic interface and get that implemented for every runtime, kinda like a libc. I don't particularly see this as realistic.

On the other hand, running wasi directly when supported or through a wrapper that does discovery means that any compliant or compatible runtime can do. We could even prompt the user to install wasmtime or wasmer via binstall before installing a wasi program, if they've got no runtime! The two big issues with a wrapper is that the wrapper itself needs to have a distribution for every platform, and wrapper updates.

This is probably still better than the next obvious thing, which is to embed a wasm runtime in Binstall, install wasi stuff to a different location like $CARGO_HOME/binstall/wasi/ and make trivial platform-dependent shell or batch scripts that essentially call exec cargo binstall --run-wasi toolname -- $*.

[NobodyXu]

Their response is about what I was expecting, and I don't really see the point of pursuing that tbh. The cool thing about wasm is it's portable, and compiling it down to a less portable version is not really the point. The compiled+dylib versions, which would be the only reasonable way to do this, would likely require tight coupling to one particular runtime and version, or to define a single dynamic interface and get that implemented for every runtime, kinda like a libc. I don't particularly see this as realistic.

Right, dylib indeed creates compatibility problems, particularly is if the users decide to upgrade wasmtime or swap the runtime.

On the other hand, running wasi directly when supported or through a wrapper that does discovery means that any compliant or compatible runtime can do. We could even prompt the user to install wasmtime or wasmer via binstall before installing a wasi program, if they've got no runtime! The two big issues with a wrapper is that the wrapper itself needs to have a distribution for every platform, and wrapper updates.

Yeah, for unix we could just use a #!/bin/sh script and for windows, well, I'm not so sure, maybe powershell?

Or we can just pre-compile a small binary for windows since almost all users of windows use x86-64 and I don't think they have ever succeeded or will succeed in convincing their customers to use their ARM windows.

Though I'm not sure how this interfere with the caching system wasmtime/wasmer has in-place.

This is probably still better than the next obvious thing, which is to embed a wasm runtime in Binstall, install wasi stuff to a different location like $CARGO_HOME/binstall/wasi/ and make trivial platform-dependent shell or batch scripts that essentially call exec cargo binstall --run-wasi toolname -- $*.

That will be the worst thing to do, since they all depend on cargo-binstall to run.

If we want to go down that path, then we probably should automatically install wasmtime/wasmer to $CARGO_HOME/bin and install wasm to $CARGO_HOME/binstall/wasi/ and then create wrapper for it in $CARGO_HOME/bin.

[NobodyXu]

@passcod I just found out that wasmer has a subcommand create-exe for creating native executable:

wasmer create-exe hello-world.wasm -o hello-world

This generates a runnable native exe on mac.

[NobodyXu]

I think that it is actually easier to support this:

If we want to go down that path, then we probably should automatically install wasmtime/wasmer to $CARGO_HOME/bin and install wasm to $CARGO_HOME/binstall/wasi/ and then create wrapper for it in $CARGO_HOME/bin.

If we want to embed a script the wasi binary, then the script will be very complex and we would have to extract the wasi to a temporary location.

Using the schema above, by installing the wasi to a special directory and create a wrapper in $CARGO_HOME/bin, we wouldn't need to extract the wasi and we could also share the script for detecting wasm runtime.

Though that means we have to rework the existing codebase and add wasi as a very special target.

[passcod]

Yeah I don't think there's any good solutions at the moment. I'd say: let's park the bulk of this issue, come back to it at some later point. It's a lot of trouble to get a fallback that's only applicable in few cases.

What we could do for now that's fairly trivial is support the case when wasi is executable directly: just plop the wasm32-wasi target in the desired target array if the detection succeeds.

[NobodyXu]

What we could do for now that's fairly trivial is support the case when wasi is executable directly: just plop the wasm32-wasi target in the desired target array if the detection succeeds.

We can do a little bit better than that by detecting binfmt_misc support using the code you wrote.

Then we just hope that windows/mac or other OSes would support something similar or they will provide a system-wide wasm runtime.

[passcod]

On the other side of the problem, it looks like we may be able to detect whether a crate is in wapm based on metadata from this tool: https://github.com/Michael-F-Bryan/cargo-wapm

# Cargo.toml
[package.metadata.wapm]
namespace = "Michael-F-Bryan"
abi = "none"

[NobodyXu]

setup-cross-toolchain-action v1.13.0 supports running wasm/wasi via binfmt

[NobodyXu]

Found a use case for wasm: a-shell on iOS/iPadOS only allows wasm to be run due to apple store policy.

https://github.com/holzschu/a-shell

[NobodyXu]

For installling wasm binaries, I think we could compile it to C using wasm2c, and then use zig-cc to compile it to an executable.

That would generate a binary without runtime dependency, with the best performance.