cross compilation to wasm32 for blst

[alxiong]

In task of #248, I encountered difficulties in cross-compiling blst crate into the wasm32-unknown-unknown target.

As time of writing, its master branch (let alone v0.3.10 which is what we originally depends on) is not no_std compliant. Thanks to the work supranational/blst#150 and alternative approach by blst's maintainer: supranational/blst#153, it's getting closer to no_std.
Unfortunately, as seen in the discussion in the latter PR, there's disagreement on how to achieve no_std and the feature flag designs, and personally I favor the former whose APIs and Rust patterns are more idiomatic, thus I created our own fork directly porting their work (full credits to Nazar and Eric) here: https://github.com/EspressoSystems/blst/tree/no-std

Native host ✅

I'm running a MacOS with m1 chip, so basically aarch64-darwin and clang from default homebrew's llvm toolchains.

Now, outside nix environment, I'm able to compile that fork of blst using: AR=$(which llvm-ar) CC=$(which clang) RUSTFLAGS="-C target-cpu=generic" cargo build --target wasm32-unknown-unknown --no-default-features

• I need explicit env var to ensure the correct choice of clang (this is most likely due to the build script is looking for $CC which I didn't set by default in my .profile)
• Since I'm running m1, the default target-cpu=apple-m13 which will cause emission of "'apple-m1' is not a recognized processor for this target (ignoring processor)" warning and some other errors. Thus, setting it to generic should be a good idea.

Inside nix-shell ❌

I got lots of help from the team, but still, it seems a bit of a hustle to switch to different clang and llvm tools and cxx headers easily. (or maybe we are not doing it correctly)

We have tried:

• https://nixos.wiki/wiki/Using_Clang_instead_of_GCC
• upgrading clang (from 11.10 to 15) inside flake.nix using this syntax here

Several problems remains:

• still need to add export CC=clang_15 in shellHook to point to the right clang (the default clang still points to the 11.1.0 version)
• with flake.nix like:

          devShell = mkShell {
            buildInputs = [
              llvmPackages_15.clang
              llvmPackages_15.libstdcxxClang
              llvm_15
            ];
           shellHook = ''
              export CC="clang-15"
              export AR="llvm-ar"
            '';

  we still got missing headers and /include files from std lib:

    running: "clang-15" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-Wall" "-Wextra" "-D__BLST_NO_ASM__" "-o" "/Users/alex/work/jellyfish/target/wasm32-unknown-unknown/debug/build/blst-f090ce078659e303/out/server.o" "-c" "/Users/alex/.cargo/git/checkouts/blst-27f1c7ed16da3054/faefc7f/bindings/rust/blst/src/server.c"
    cargo:warning=In file included from /Users/alex/.cargo/git/checkouts/blst-27f1c7ed16da3054/faefc7f/bindings/rust/blst/src/server.c:7:
    cargo:warning=In file included from /Users/alex/.cargo/git/checkouts/blst-27f1c7ed16da3054/faefc7f/bindings/rust/blst/src/keygen.c:7:
    cargo:warning=In file included from /Users/alex/.cargo/git/checkouts/blst-27f1c7ed16da3054/faefc7f/bindings/rust/blst/src/consts.h:8:
    cargo:warning=/Users/alex/.cargo/git/checkouts/blst-27f1c7ed16da3054/faefc7f/bindings/rust/blst/src/vect.h:9:10: fatal error: 'stddef.h' file not found
    cargo:warning=#include <stddef.h>
    cargo:warning=         ^~~~~~~~~~
    cargo:warning=1 error generated.
    exit status: 1
  

What now?

We use target-specific conditional dependency as a temporary solution, only use blst in non-wasm env, and use this issue to track progress on blst's wasm32 inside nix in future work.

cc @ChaitanyaKonda @DieracDelta @jbearer @nyospe @nomaxg @mike1729

[alxiong]

thanks to @sveitser, we now have a solution inside nix-shell as well 🥂 :

with import <nixpkgs> {
    overlays = [
        (self: super: {
            rustc = (super.rustc.override {
                stdenv = self.stdenv.override {
                    targetPlatform = super.stdenv.targetPlatform // {
                        parsed = {
                            cpu = { name = "wasm32"; };
                            vendor = {name = "unknown";};
                            kernel = {name = "unknown";};
                            abi = {name = "unknown";};
                        };
                    };
                };
            }).overrideAttrs (attrs: {
                configureFlags = attrs.configureFlags ++ ["--set=build.docs=false"];
            });
        })
    ];
};

clang15Stdenv.mkDerivation {
  name = "clang-nix-shell";
  buildInputs = [
    cargo
    rustc
    clang-tools_15
    clangStdenv
    llvm_15
  ];
  shellHook = ''
    export C_INCLUDE_PATH="${llvmPackages_15.libclang.lib}/lib/clang/15.0.7/include"
    export CC=$(which clang)
    export AR=$(which llvm-ar)
    export CFLAGS="-mcpu=generic"
  '';
}

---

One little caveats for m1 is that somehow RUSTFLAGS="-C target-cpu=generic" doesn't propagate the choice of CPU to clang, thus I need to add extra CFLAGS explicitly on the last line of flake.nix snippet above.