Crate fails to link for mips-unknown-linux-musl target with mips-linux-gnu-gcc compiler

[tymcauley]

Summary

When building a statically-linked Rust binary crate for the mips-unknown-linux-musl target, linking a crate fails with the following error:

error: linking with `/opt/musl/bin/musl-gcc` failed: exit code: 1
...
  = note: /home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/libstd-0c86013d1a999471.rlib(elf.o): In function `elf_zlib_fetch':
          /checkout/src/libstd/../libbacktrace/elf.c:1097: undefined reference to `__bswapsi2'
...
          collect2: error: ld returned 1 exit status

Expected result: the crate successfully compiles.

Actual result: the crate fails to compile, with an error during the linking stage.

Platform: Ubuntu 18.04, x86_64.

Output of rustc --version --verbose:

rustc 1.32.0 (9fda7c223 2019-01-16)
binary: rustc
commit-hash: 9fda7c2237db910e41d6a712e9a2139b352e558b
commit-date: 2019-01-16
host: x86_64-unknown-linux-gnu
release: 1.32.0
LLVM version: 8.0

MUSL standard C library version: 1.1.20.

C compiler: MIPS GCC version 7.3, provided by Ubuntu 18.04 (gcc-mips-linux-gnu).

I tried searching the open and closed issues for this project, and couldn't find any others exhibiting this same error.

More Detail

Forgive me if this is the wrong place to file this bug, but I'm not entirely sure how to debug this problem, so I'm starting here with Rust.

The file elf.c in libbacktrace makes a call to __builtin_bswap32 on big-endian hosts (see source code here). This function gets translated into the symbol __bswapsi2 for the MIPS target, and should be present in the compiler's built-in library, but it appears it's absent here.

Now, I can see that the Rust CI build for the MIPS MUSL target uses GCC 5.3 from the OpenWRT toolchain. I'm not entirely sure why this should give any different results from using the GCC 7.3 package provided by Ubuntu 18.04 (gcc-mips-linux-gnu). I've also tried this experiment using the gcc-5-mips-linux-gnu Ubuntu package, which provides GCC 5.5, and the results are the same (fails to link due to undefined reference to __bswapsi2).

Here is the full error message from trying to build the crate:

   Compiling hello-world v0.1.0 (/home/rust-user/hello-world)
     Running `rustc --edition=2018 --crate-name hello_world src/main.rs --color never --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=6ade4216daf7af7c -C extra-filename=-6ade4216daf7af7c --out-dir /home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/deps --target mips-unknown-linux-musl -C linker=/opt/musl/bin/musl-gcc -C incremental=/home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/incremental -L dependency=/home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/deps -L dependency=/home/rust-user/hello-world/target/debug/deps -C target-feature=+crt-static`
error: linking with `/opt/musl/bin/musl-gcc` failed: exit code: 1
  |
  = note: "/opt/musl/bin/musl-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-nostdlib" "/home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/crt1.o" "/home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/crti.o" "-L" "/home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib" "/home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/deps/hello_world-6ade4216daf7af7c.1h9cu8y3f5q3d4un.rcgu.o" "/home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/deps/hello_world-6ade4216daf7af7c.2kefx4d4gnwor8e3.rcgu.o" "/home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/deps/hello_world-6ade4216daf7af7c.3om4rf8tfbswki2o.rcgu.o" "/home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/deps/hello_world-6ade4216daf7af7c.4cb9yoeympkwffqu.rcgu.o" "/home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/deps/hello_world-6ade4216daf7af7c.4i47wsei6k926btj.rcgu.o" "/home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/deps/hello_world-6ade4216daf7af7c.4pjni4voqjrzsg20.rcgu.o" "-o" "/home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/deps/hello_world-6ade4216daf7af7c" "/home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/deps/hello_world-6ade4216daf7af7c.yzibwqqnr0tk21b.rcgu.o" "-Wl,--gc-sections" "-no-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/deps" "-L" "/home/rust-user/hello-world/target/debug/deps" "-L" "/home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/libstd-0c86013d1a999471.rlib" "/home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/libpanic_unwind-e04840eda0100e05.rlib" "/home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/libunwind-16bfb4d182748e63.rlib" "/home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/liblibc-185d9b81685a2f08.rlib" "/home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/liballoc-4806247fe4c34e64.rlib" "/home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/libcore-879310dc3b96af61.rlib" "-Wl,--end-group" "/home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/libcompiler_builtins-d0572f7a936161bf.rlib" "-static" "-Wl,-Bdynamic" "/home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/crtn.o"
  = note: /home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/libstd-0c86013d1a999471.rlib(elf.o): In function `elf_zlib_fetch':
          /checkout/src/libstd/../libbacktrace/elf.c:1097: undefined reference to `__bswapsi2'
          /checkout/src/libstd/../libbacktrace/elf.c:1097: undefined reference to `__bswapsi2'
          /checkout/src/libstd/../libbacktrace/elf.c:1097: undefined reference to `__bswapsi2'
          /checkout/src/libstd/../libbacktrace/elf.c:1097: undefined reference to `__bswapsi2'
          /checkout/src/libstd/../libbacktrace/elf.c:1097: undefined reference to `__bswapsi2'
          /home/rust-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/mips-unknown-linux-musl/lib/libstd-0c86013d1a999471.rlib(elf.o):/checkout/src/libstd/../libbacktrace/elf.c:1097: more undefined references to `__bswapsi2' follow
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

error: Could not compile `hello-world`.

Caused by:
  process didn't exit successfully: `rustc --edition=2018 --crate-name hello_world src/main.rs --color never --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=6ade4216daf7af7c -C extra-filename=-6ade4216daf7af7c --out-dir /home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/deps --target mips-unknown-linux-musl -C linker=/opt/musl/bin/musl-gcc -C incremental=/home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/incremental -L dependency=/home/rust-user/hello-world/target/mips-unknown-linux-musl/debug/deps -L dependency=/home/rust-user/hello-world/target/debug/deps -C target-feature=+crt-static` (exit code: 1)

How to Reproduce

An easy way to reproduce this is to create a Dockerfile with the following contents, and then build it with docker build .:

FROM ubuntu:18.04
ENV USERNAME rust-user
ENV MUSL musl-1.1.20

RUN apt update && apt install -y \
    build-essential \
    curl \
    gcc-mips-linux-gnu

# Install MUSL for MIPS.
WORKDIR /tmp
RUN curl https://www.musl-libc.org/releases/$MUSL.tar.gz | tar xzf - && \
    cd $MUSL && \
    CC=mips-linux-gnu-gcc ./configure --prefix=/opt/musl --enable-wrappers=all && \
    make -j install

RUN useradd -m $USERNAME && (echo "$USERNAME:$USERNAME" | chpasswd)
USER $USERNAME
WORKDIR /home/$USERNAME

# Install Rust.
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
ENV PATH="/home/$USERNAME/.cargo/bin:$PATH"
RUN rustup target add mips-unknown-linux-musl

# Create and build test project.
RUN cargo new hello-world
WORKDIR /home/$USERNAME/hello-world
RUN mkdir .cargo && \
    echo '[target.mips-unknown-linux-musl]'                 >> .cargo/config && \
    echo 'linker = "/opt/musl/bin/musl-gcc"'                >> .cargo/config && \
    echo 'rustflags = ["-C", "target-feature=+crt-static"]' >> .cargo/config
RUN cargo build --verbose --target mips-unknown-linux-musl

[nagisa]

The necessary functions should be added to the compiler-builtins.

[tymcauley]

This does look like an interesting idea, but when trying to compile with that crate, I get lots of errors of this sort:

error[E0554]: #![feature] may not be used on the stable release channel
  --> /home/rust-user/.cargo/git/checkouts/compiler-builtins-ec094dc45a0179c8/fcdf1bc/src/lib.rs:11:1
   |
11 | #![feature(asm)]
   | ^^^^^^^^^^^^^^^^

You should be able to recreate this by adding the following line to the Dockerfile I mentioned in the original issue, just before the cargo build line at the end:

RUN echo 'compiler_builtins = { git = "https://github.com/rust-lang-nursery/compiler-builtins" }' >> Cargo.toml

I'm currently working on a project that I'd like to keep on the stable channel. Will this crate not work at all on the stable channel?

[mati865]

@rustbot modify labels: +O-musl

[jesskfullwood]

possibly related to #61328

[glebpom]

Any updates with this issue? Have the same problem

[draganmladjenovic]

Adding "-C", "link-args=-lgcc" to re-add libgcc removed by -nostdlib to rustargs fixes the problem, at least for the simple hello world. Or you could implement __bswapsi2 yourself. I'm guessing that openwrt toolchain targets mips32 as opposed of mips32r2, so it requires "libcall" for bswap. Hope it helps.

[glebpom]

It works, thanks!