*-musl cross builds fail on proc_macro

[drozdziak1]

Steps to reproduce

The issue happens in the proc-macro2 crate on which we indirectly depend. serde depends on it, so we probably won't get away with "dependency amputation". For just reproducing the issue:

1. Enter althea-firmware
2. Open roles/prepare-config/defaults/main.yml
3. Swap out the entry in feeds_list to src-git althea https://github.com/drozdziak1/althea-packages.git;proc-macro-debug-trash
4. Start the build with any of the defined targets - ansible-playbook should exit with an error
5. cd build/ to investigate
6. Rebuild with make -j1 package/althea-rust-binaries/compile V=s to see the full build output
   Steps 1-5. need to be done only once and as long as you don't swap out the althea-pachages to use you only need to do 6. in order to rebuild.

To experiment with our OpenWrt package definition Makefile (This requires doing the steps from the list above):

1. Clone this and check out branch proc-macro-debug-trash
2. Get back to althea-firmware and again open roles/prepare-config/defaults/main.yml
3. This time swap the entry out for src-link althea </full/path/to/where/you/cloned/althea-packages>
4. cd build/
5. scripts/feeds uninstall althea-rust-binaries
6. scripts/feeds uninstall althea-babeld
7. scripts/feeds uninstall generate-ipv6-address
8. cd .. back to althea-firmware root
9. do steps 4-6 from the first list
   Now changes in althea/althea-rust-binaries/Makefile should be immediately applied to the firmware build upon rebuilding (step 6. from the first list in althea-firmware/build)

To experiment with althea_rs in the cross build (This requires doing the steps from the list above):

1. Clone this
2. git checkout -b proc-macro-debug-trash to create a disposable trash branch to experiment in
3. cd into the path where your copy of althea-packages checked out at proc-macro-debug-trash is
4. open althea/althea-rust-binaries/Makefile
5. Change PKG_SOURCE_VERSION to proc-macro-debug-trash
6. Change PKG_SOURCE_URL to the full path to the copy of althea_rs that you just cloned
7. Do steps 4-6 from the first list
   Now althea-firmware should've built althea_rs using the copy you cloned in step 1. After making a change in your local althea_rs commit it to the proc-macro-debug-trash branch and do rm -rf dl/althea-rust-binaries* in althea-firmware/build. Now you can rebuild (step 6 from the first list) and see the effect.

Expected result

The build succeeds and installs rita and guac-light-client in the firmware

What's actually happening

The cross build for *-musl targets fails because a proc_macro crate cannot be found, presently while building a proc-macro2 crate.

The underlying issue

This seems to be a flaw in Rust itself as only musl targets are ever affected regardless of whether we use OpenWrt or not or whether we build althea_rs or anything else that uses proc_macro (check out the minimal test case below). The problem is that the procedural macro crate is a dynamically linked library. For some reason this messes up builds against the musl libc implementation. See Links for more information.

Minimal test case

Apart from the relatively complicated reproduction steps above, it's possible to trivially reproduce the underlying issue outside the OpenWrt cross-build:

1. Create a new cargo library project with cargo new blah
2. Open blah/src/lib.rs and add extern crate proc_macro; on top
3. Run cargo build --target <some -musl target>

The build should fail on proc_macro being not found and at the same time compile normally without --target <some -musl target> or a -gnu target. If the proc_macro issue was not present, the build should complain about the linker being unspecified. OpenWrt deals with this by supplying all necessary cross environment information through env variables.

Links

• Ongoing discussion in proc-macro2 (the crate where the actual error appears) Build fails on -musl targets because of proc_macro dtolnay/proc-macro2#52
• A similar closed issue in the official Rust repo: Unable to find crate proc_macro on musl target rust-lang/rust#40174

[drozdziak1]

There's several workarounds I've been thinking about:

• Switch to GNU libc - There's issues in OpenWrt itself that surface when we diverge from the default build configuration and switch the libc to glibc in menuconfig
• ditch the dependencies that touch the problematic packages - a very short-sighted solution as important packages like serde depend on packages which use the procedural macro features

[drozdziak1]

libc switch update

I managed to get our binaries to build, however the proc_macro feature still seems to be causing problems. Without our package, the glibc build continues and successfully finishes for VirtualBox.

The new problem with libstd-[some_short_hash].so

A new issue that surfaced is a weird new linking dependency that guac-light-client now has. It's a library called libstd-[some_short_hash] where some_short_hash is a bunch of hex digits. This dependency makes the OpenWrt build fail due to dependencies unspecified in the package Makefile (it automatically does a check with ldd on the binaries after compilation). Some extra facts:

• The *.so itself lives somewhere within ~/.rustup (found it with locate)
• As file indicates, the *.so is a cross object file for 32-bit x86 in case of Virtualbox
• This issue suggests that the problem may have something to do with the dynamic linking of proc_macro (maybe a custom Rust libstd is made for the occasion?)

Possible solutions

• Given that the *.so is meant for the cross environment, maybe we could design a reliable way to extract it and install it with the binaries

Links

• A description of the procedural macro functionality - It links to the relevant RFC and GitHub issue which should help with understanding the implementation

[drozdziak1]

Fun fact: The problematic crate (proc-macro2) along with the crate immediately dependent on it (quote) don't seem to follow what https://doc.rust-lang.org/nightly/unstable-book/language-features/proc-macro.html lists as mandatory for proc macro crates to work (no proc-macro = true, no #![feature(proc_macro)]). What's weird is that both quote and proc-macro2 compile okay unless we try to compile a crate that depends on quote - then it fails on cross builds

[jkilpatr]

So to be clear we added proc-macro true to one of our crates and it fixed the proc-macro2 issue? Now that I've successfully built for mips I'm trying to correlate all the facts.

[drozdziak1]

I did not add proc-macro = true anywhere. I usually list explicitly the stuff that worked for me.

[drozdziak1]

@jkilpatr Pasting the script that worked for you (the version you posted on Riot) so it doesn't get lost

#!/bin/bash
export CARGO_TARGET_MIPS_UNKNOWN_LINUX_MUSL_LINKER=/home/justin/repos/althea-firmware/build/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/bin/mips-openwrt-linux-gcc
export TARGET_CC=/home/justin/repos/althea-firmware/build/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/bin/mips-openwrt-linux-gcc
export HOST_CC=gcc
export OPENSSL_STATIC=true
export MIPS_UNKNOWN_LINUX_MUSL_OPENSSL_DIR=/home/justin/repos/althea-firmware/build/staging_dir/target-mips_24kc_musl/usr/
export PKG_CONFIG_ALLOW_CROSS=1

#cargo clean
cargo build --target mips-unknown-linux-musl --release

[drozdziak1]

You could technically say that this issue is resolved as it doesn't get in our way anymore, but I think we'll see more of it in the future.