linking with cc failed on AArch64 with aarch64-unknown-linux-musl target

[MrXinWang]

I was trying to build a rust-based binary with cargo in docker container on AArch64, however an undefined reference to symbol error occurs when using musl target. Any help would be greatly appreciated, thanks!

Note that the binary can be successfully built with GNU toolchain on AArch64, and also the AArch64 binary can be successfully cross-built with both GNU and musl toolchain on x86_64.

Building Environment:
A docker container from this Dockerfile.

# git clone https://github.com/cloud-hypervisor/cloud-hypervisor.git

# cd cloud-hypervisor

# git checkout 096ffe08f244fa0d35bc44a3987a671ffe5b83a4

# ./scripts/dev_cli.sh build --release --libc musl

Rust toolchain version:
1.43.0

Building command:
RUSTFLAGS="-C link_arg=-lgcc" cargo build --release --target aarch64-unknown-linux-musl --no-default-features --features mmio

The reason why adding link_arg=-lgcc is to resolve #46651 (comment)

Error log:

error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-nostdlib" "/root/.rustup/toolchains/1.42.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/crt1.o" "/root/.rustup/toolchains/1.42.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/crti.o" "-L" "/root/.rustup/toolchains/1.42.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib" "/root/cloud-hypervisor-dev/cloud-hypervisor-ci/target/aarch64-unknown-linux-musl/release/deps/cloud_hypervisor-7337885b601979f6.cloud_hypervisor.4uvd3xas-cgu.0.rcgu.o" "-o" "/root/cloud-hypervisor-dev/cloud-hypervisor-ci/target/aarch64-unknown-linux-musl/release/deps/cloud_hypervisor-7337885b601979f6" "-Wl,--gc-sections" "-no-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/root/cloud-hypervisor-dev/cloud-hypervisor-ci/target/aarch64-unknown-linux-musl/release/deps" "-L" "/root/cloud-hypervisor-dev/cloud-hypervisor-ci/target/release/deps" "-L" "/root/.rustup/toolchains/1.42.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/tmp/rustcQSeDtF/libbacktrace_sys-f3af7a69ff7b3bd7.rlib" "/tmp/rustcQSeDtF/libunwind-3ab3605e5327960d.rlib" "/tmp/rustcQSeDtF/liblibc-1170e40c1719a823.rlib" "/root/.rustup/toolchains/1.42.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-cb2050f8f4e1771d.rlib" "-Wl,-Bdynamic" "-lfdt" "-static" "-lgcc" "/root/.rustup/toolchains/1.42.0-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/crtn.o"
  = note: /usr/bin/ld: /tmp/rustcQSeDtF/liblibc-1170e40c1719a823.rlib(__stack_chk_fail.lo): undefined reference to symbol '__stack_chk_guard@@GLIBC_2.17'
          /usr/bin/ld: /lib/aarch64-linux-gnu/ld-linux-aarch64.so.1: error adding symbols: DSO missing from command line
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

[JamieCunliffe]

I have had a look into this and I have managed to get it building with musl.

I think you have to add the musl-gcc specs file to the link_arg, as you are running this on a glibc distro.

RUSTFLAGS="-C link_arg=-lgcc -C link_arg=-specs -C link_arg=/usr/lib/aarch64-linux-musl/musl-gcc.specs" cargo build --release --target aarch64-unknown-linux-musl --no-default-features --features mmio

Once I did this I started getting errors that libfdt couldn't be found, when I built libfdt using musl-gcc and copied libfdt to /usr/lib/aarch64-linux-musl then the build succeeded.

I might be missing something here but I can't see where RUSTFLAGS are being set in that linked cross-build.yml so I'm not fully sure what's going on there.

[mati865]

What if you try to use AArch64 linker instead?
RUSTFLAGS="-C linker=aarch64-linux-gnu-gcc cargo build --release --target aarch64-unknown-linux-musl --no-default-features --features mmio

[MrXinWang]

Hi @JamieCunliffe Great thanks for your effort! It works perfectly! I probably need to think about the reason why it works. If there is anything else, I may probably contact you :)

What if you try to use AArch64 linker instead?
RUSTFLAGS="-C linker=aarch64-linux-gnu-gcc cargo build --release --target aarch64-unknown-linux-musl --no-default-features --features mmio

@mati865 I also tried your method but sadly this cannot work :( Thanks for your reply anyway ^^

[MrXinWang]

Closing this issue as it has been fixed.