Add support for ARM

[ethanfrey]

This is now supported in wasmer (as of 0.11) and has been requested by Zach. If we can update the build-scripts to support it, that would be a cool addition

https://medium.com/wasmer/running-webassembly-on-arm-7d365ed0e50c

[ethanfrey]

Update: I tried but there were still some issues with wasmer. Also, they are going through a re-write to 1.0, so not worth the time now.

I would ask for help on this for some Pi-lover once we have a solid wasmer release.

[faddat]

Well I am certainly pi lover, so let me know when you think we have a solid wasmer release 👍

[webmaster128]

Wasmer claims to have good support for the new ARM based MacBook now. So it should be possible to support both aarch64-apple-darwin and aarch64 Linux.

[Kwaskoff]

Well I am certainly pi lover, so let me know when you think we have a solid wasmer release +1

We got Wasmer release already!
https://github.com/wasmerio/wasmer/releases/tag/1.0.1

[webmaster128]

@Kwaskoff was there any relevant change in Wasmer between 1.0.0 (integrated here already) and 1.0.1?

This ticket is unblocked but the Confio team will probably not work on it until we have developers with ARM Macs as dev machines. If this is important to you, feel free to look into it. A good starting point is the compatibility table here: https://github.com/CosmWasm/wasmvm#overview

[faddat]

I... believe that this is solvable.

[faddat]

Graviton2, Ubuntu 20.10:

sudo apt install llvm-11-dev llvm-11
export IS_LINUX=1
export IS_AARCH64=1

What if it was not the singlepass compiler?

May I use cranelift and expect it to work?

[faddat]

#####
#
# The Matrix
#
#####

# The matrix is the product of the following columns:
#
# |------------|--------|----------|--------------|-------|
# | Compiler   ⨯ Engine ⨯ Platform ⨯ Architecture ⨯ libc  |
# |------------|--------|----------|--------------|-------|
# | Cranelift  | JIT    | Linux    | amd64        | glibc |
# | LLVM       | Native | Darwin   | aarch64      | musl  |
# | Singlepass |        | Windows  |              |       |
# |------------|--------|----------|--------------|-------|
#
# Here is what works and what doesn't:
#
# * Cranelift with the JIT engine works everywhere,
#
# * Cranelift with the Native engine works on Linux+Darwin/`amd64`,
#   but it doesn't work on */`aarch64` or Windows/*.
#
# * LLVM with the JIT engine works on Linux+Darwin/`amd64`,
#   but it doesn't work on */`aarch64` or Windows/*.
#
# * LLVM with the Native engine works on
#   Linux+Darwin/`amd64`+`aarch64`, but it doesn't work on Windows/*.
#
# * Singlepass with the JIT engine works on Linux+Darwin/`amd64`, but
#   it doesn't work on */`aarch64` or Windows/*.
#
# * Singlepass with the Native engine doesn't work because it doesn't
#   know how to output object files for the moment.
#
# * Windows isn't tested on `aarch64`, that's why we consider it's not
#   working, but it might possibly be.

[ethanfrey]

Cranelift compiles better. BUT it is not safe for a blockchain context.
It is an optimizing compiler and it is expected that giving it arbitrary malicious wasm to compile could open up attack vectors, minimally O(N^2) compile time (where N is the size of the wasm contract)

We need this to compile with singlepass to be useful for wasmd

[faddat]

"Cranelift with the JIT engine" is OK?

[ethanfrey]

If this is not supported by wasmer, feel free to jump on their github and raise an issue. (It will have more chance of getting addressed)

[ethanfrey]

All design and testing has been with singlepass. It is the only engine we feel comfortable with using in a blockchain setting.

[faddat]

All design and testing has been with singlepass. It is the only engine we feel comfortable with using in a blockchain setting.

dang.

LLVM is a no-go for the same reasons that Cranelift is a no-go?

[ethanfrey]

• Singlepass with the JIT engine works on Linux+Darwin/amd64, but
  it doesn't work on /aarch64 or Windows/.

Thus the long-open tickets to add arm64 and windows support.

Every time someone says it's working in wasmer now, it turns out it is a different backend.

[ethanfrey]

LLVM is a no-go for the same reasons that Cranelift is a no-go?

LLVM has blazing fast code execution (I clocked wasm doing loops of sha256 pre-compiled with LLVM but inside the wasmer vm to be within a factor of two of the same rust code compiled directly to native binary). However, the compilation times are long (think close to 1 second) and surely have far, far more compiler bombs lurking than cranelift. I benchmarked it in the beginning, but discarded it for use in a blockchain setting shortly afterwards

[faddat]

got it. Ok, so to fix this (I'm pretty OK w/ arm ports....) I would need to make "singlepass with the JIT engine" work on arm64, yes?

https://developers.redhat.com/blog/2020/01/20/mir-a-lightweight-jit-compiler-project/

"use in a blockchain context" == determinism concerns?

https://medium.com/nearprotocol/wasm-for-blockchain-2019-d093bfeb6133

I'm getting this feeling you've read both articles, however.

Or possibly mir == the JIT compiler mentioned in the config.

https://github.com/rust-lang/miri

wasmerio/wasmer#713

[faddat]

NEAR seems to be using LLVM

Too scary?

Perhaps they've optimized?

[faddat]

https://github.com/wasmerio/wasmer/tree/63a2d8129c7ad5ca670d041859de45e01292dd12/lib/singlepass-backend/src

In the past, they had a translator for the x86 asm instructions in the emitter.

It is my understanding that codegen is often used to great effect in such cases. Not sure if that'll work here.

https://censoredusername.github.io/dynasm-rs/language/index.html

[ethanfrey]

NEAR seems to be using LLVM
Too scary?
Perhaps they've optimized?

I'd be happy to see a link of what they are doing.

That said, they have some very skilled developers building a new platform from the ground up and I have a lot of respect for their engineering. If they use LLVM, they probably manage to run the compilation in some "background process" / "offchain worker" and sandbox it somehow to avoid run-away processs.

I have considered doing some hybrid approach, where it is available as singlepass right away and some selected contracts get recompiled in LLVM in an offchain worker. But there is no such infrastructure available for the Cosmos SDK and I didn't feel like inventing yet another subsystem.

[faddat]

https://spectrum.chat/wasmer/general/wasmer-aarch64-support~58dd1879-3379-42a4-94ce-1e1212f2efcc

[ethanfrey]

"use in a blockchain context" == determinism concerns?

Kind of (which is why we ban some floats). But mainly it is ability to gas meter all code execution or have a predictable performance. Singlepass gives us predictable O(N) compilation time, so we can just charge based on wasm bytecode size.

Optimizing compilers provide neither and have no upper limit on execution time (which blocks all other processing on every full node in the blockchain). This could be an extremely dangerous attack vector.

[ethanfrey]

If you want to work on adding arm64 bytecode emission to wasmer, that could be a fun rust project (IIFC, you are skilled in Rust) and a nice help to the ecosystem.

If your employer would give you time for that...

[faddat]

Yeah, this seems to affect many projects that want ARM64 build targets and CosmWasm

• Desmos
• Stargaze
• Splash
• Juno

Alternative is a little silly: x86_64 single board computers in place of Raspberry Pi's

I am pretty solidly booked, time-wise, but knew I'd be up insanely late tonight as I'm going on a podcast. Figured that I could bang this out.

Perhaps not though.

The hopeful bit is:

the emitter uses dynasm and dynasm fully supports aarch64.

I'm not even remotely skilled in Rust, yet.

[ethanfrey]

If you really want to dig in, there was a magic streaming/optimizing compiler that produced highly efficient bytecode in a singlepass.

At least this is what Parity claimed: https://www.parity.io/lightbeam-webassembly-compiler/

I'm not sure what the current state is, but you can find the code here: https://github.com/bytecodealliance/wasmtime/tree/main/crates/lightbeam

Maybe someone could get this working with arm64 and integrated with wasmer 🤯

[faddat]

Somehow, I figure that humans aren't writing these 7-8k LOC files like codegen_x86.rs

Could be wrong though, people have done stranger things :P

This won't get finished tonight, that's for sure.

@ethanfrey thanks for helping me understand the issues.

[faddat]

@ethanfrey TBH I don't know how much I truly want to dig in.

Just, I want this on my Pi's.

Could you let me know what you think?

https://crates.io/crates/wasmer-vm-near/reverse_dependencies

[webmaster128]

THERE IS NO ARM SUPPORT IN SINGLEPASS, PERIOD.

[antrixy]

Will this be resvoled soon. I am raspberry lover. Its been more than two years since this issue was opened.

[0xjame5]

@webmaster128 could you clarify more, do you mean indefinitely there will be no support for Arm based development using wasmnvm?

[ethanfrey]

@webmaster128 could you clarify more, do you mean indefinitely there will be no support for Arm based development using wasmnvm?

Please read the whole thread. Like this #53 (comment)

There is currently no upstream support for arm64 in wasmer singlepass compiler.

This is our largest request for support from Wasmer and can be tracked here https://github.com/wasmerio/wasmer/labels/project-confio

This is a known issue and hard to solve and we are patiently awaiting upstream.

[faddat]

I'd like to second @ethanfrey here:

I spent a lot of time trying to fix this at one point, that is why this issue's thread is so long. It is very difficult and the fix is likely best done upstream.

[mandrean]

FYI there's a release candidate of wasmer with singlepass arm64 support now!

https://github.com/wasmerio/wasmer/releases/tag/2.2.0-rc1

[shanev]

There's now an RC2: https://github.com/wasmerio/wasmer/releases/tag/2.2.0-rc2. Almost there!

[webmaster128]

The integration PR for cosmwasm-vm is here: CosmWasm/cosmwasm#1224.

Please don't get too excited about this yet. We'll have to create a build system, which more or less doubles the number of builds we create. And every time 1 becomes n, you create a new dimension and things get harder in a non-trivial way. It might even be blocked by #277 but I'm not sure about that. We might get away with Rust 1.55.0 even if not officially supported by the Wasmer team.

[mandrean]

Please don't get too excited about this yet

IMHO what most people are excited about is running local dev environments on their new M1 Macs. Some quirks and bugs are acceptable. Being able to use your new expensive laptop is the biggest win

I've actually been doing this with a fork of wasmvm and cosmwasm to get terrad to run natively on my laptop since ~1 month back:

mandrean/terra-core@aebba18
mandrean@b9e4201
mandrean/cosmwasm@90cb117

Now, production-worthy arm64 for Linux VMs in the cloud is a different story. Maybe that will take a longer time to sort out. But this is already great stuff for devs with new Macbooks.

[webmaster128]

This table should capture where we are and where we go with the build system.

OS family	Arch	Linking	Supported	Wasmer 2.2+	Note
Linux (glibc)	x86_64	shared	✅​libwasmvm.so	✅​libwasmvm.so
Linux (glibc)	x86_64	static	🚫​	🚫​	Would link libwasmvm statically but glibc dynamically as static glibc linking is not recommended. Potentially interesting for Osmosis.
Linux (glibc)	aarch64	shared	🚫​	✅​libwasmvm.aarch64.so
Linux (glibc)	aarch64	static	🚫​	🚫​
Linux (musl)	x86_64	shared	🚫​	🚫​	Possible but not needed
Linux (musl)	x86_64	static	✅​libwasmvm_muslc.a	✅​libwasmvm_muslc.a
Linux (musl)	aarch64	shared	🚫​	🚫​	Possible but not needed
Linux (musl)	aarch64	static	🚫​	✅​libwasmvm_muslc.aarch64.a
macOS	x86_64	shared	✅​libwasmvm.dylib	✅​libwasmvm.dylib	Fat/universal library can contain multiple archs
macOS	x86_64	static	🚫​	🚫​
macOS	aarch64	shared	🚫​	✅​libwasmvm.dylib	Fat/universal library can contain multiple archs
macOS	aarch64	static	🚫​	🚫​
Windows (mingw)	x86_64	shared	🏗 wasmvm.dll	🏗 wasmvm.dll	See #288
Windows (mingw)	x86_64	static	🚫​	🚫​
Windows (mingw)	aarch64	shared	🚫​	🚫​
Windows (mingw)	aarch64	static	🚫​	🚫​

Turns out that it is possible to have two architectures in a .dylib at the same time. So we can start with M1 support relatively easily as this does not add new filenames to distinguish builds.

[webmaster128]

mandrean/terra-core@aebba18
mandrean@b9e4201
mandrean/cosmwasm@90cb117

Interesting approach to focus on aarch64 Linux and then use that for M1.

[webmaster128]

#296 shows this universal library for macOS is working well.

[webmaster128]

The good news is, we have an libwasmvm_muslc.aarch64.a now. The bad news is, it's not working. If anyone wants to try it in their chain or understands the error message, I'd be happy to discuss in #306.

[webmaster128]

libwasmvm_muslc.aarch64.a is shipped as part of v1.0.0-beta9 and is working when the linker flags are set correctly.

[webmaster128]

See https://github.com/CosmWasm/wasmvm/wiki/Linking-wasmvm_muslc for how to use the new static libraries.

[migueldingli1997]

What's the latest status around this please?

[webmaster128]

What's the latest status around this please?

We have aarch64 support (64 bit ARM) done and released. So I think this ticket can be closed. Is there anything you are missing?

[Kwaskoff]

What's the latest status around this please?

We have aarch64 support (64 bit ARM) done and released. So I think this ticket can be closed. Is there anything you are missing?
I can check it out this month

[webmaster128]

ARM support is done for now. I'd not recommend using it for consensus critical infrastructure yet. There was some issue popping up here: CosmWasm/cosmwasm#1628. We can follow up with more specific tickets.