pkgsStatic.llvmPackages.libcxxStdenv does not produce static c++ binaries

[tfc]

Describe the bug

Using pkgs.pkgsStatic, it is easily possible to compile static c++ binaries using -static.
The same does not seem to work if stdenv = pkgsStatic.llvmPackages.libcxxStdenv;.

To Reproduce

Minimal working example:

# static.nix
let
  mainCpp = builtins.toFile "main.cpp" ''
    #include <iostream>
    int main() { std::cout << "hello\n"; }
  '';

  deriv = { stdenv }:
    stdenv.mkDerivation {
      name = "hello-app";
      src = mainCpp;
      unpackPhase = ":";
      buildPhase = "$CXX $src -o hello -static";
      installPhase = "mkdir -p $out/bin && install -m755 hello $out/bin/hello";
    };
  overlay = self: super: {
    hello-app = self.callPackage deriv {};
  };
  nixpkgs = builtins.fetchTarball {
    url = "https://github.com/nixos/nixpkgs/archive/f217c0ea7c148ddc0103347051555c7c252dcafb.tar.gz";
    sha256 = "0cyksxg2lnzxd0pss09rmmk2c2axz0lf9wvgvfng59nwf8dpq2kf";
  };
  pkgs = (import nixpkgs { overlays = [ overlay ]; }).pkgsStatic;
in
{
  hello-static-gcc = pkgs.hello-app;
  hello-static-clang = pkgs.hello-app.override {
    stdenv = pkgs.llvmPackages.libcxxStdenv;
  };
}

the gcc derivation works well:

$ ldd $(nix-build static.nix -A hello-static-gcc)/bin/hello
	not a dynamic executable

With clang, the following happens:

$ nix-build static.nix -A hello-static-clang
# ... lots of output omitted ...
(.text+0x2b4): undefined reference to `__cxa_guard_acquire'
/nix/store/2v3vkva29310005gipcqhdw9hxr57j6l-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ld: (.text+0x2f2): undefined reference to `__cxa_guard_release'
/nix/store/2v3vkva29310005gipcqhdw9hxr57j6l-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/s1rdpf0z4l4d0m5rsysjl0fna2ayf3db-libc++-7.1.0-x86_64-unknown-linux-musl/lib/libc++.a(future.cpp.o): in function `std::__1::future_error::~future_error()':
(.text+0x2f): undefined reference to `std::logic_error::~logic_error()'
/nix/store/2v3vkva29310005gipcqhdw9hxr57j6l-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/s1rdpf0z4l4d0m5rsysjl0fna2ayf3db-libc++-7.1.0-x86_64-unknown-linux-musl/lib/libc++.a(future.cpp.o):(.data.rel.ro._ZTINSt3__112future_errorE[_ZTINSt3__112future_errorE]+0x10): undefined reference to `typeinfo for std::logic_error'
/nix/store/2v3vkva29310005gipcqhdw9hxr57j6l-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/s1rdpf0z4l4d0m5rsysjl0fna2ayf3db-libc++-7.1.0-x86_64-unknown-linux-musl/lib/libc++.a(future.cpp.o):(.data.rel.ro._ZTVNSt3__112future_errorE[_ZTVNSt3__112future_errorE]+0x20): undefined reference to `std::logic_error::what() const'
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
builder for '/nix/store/h89wg44nxqpf5fxbzkq4lykv4l19rlc3-hello-app-x86_64-unknown-linux-musl.drv' failed with exit code 1
error: build of '/nix/store/h89wg44nxqpf5fxbzkq4lykv4l19rlc3-hello-app-x86_64-unknown-linux-musl.drv' failed

Looking at pkgs/development/compilers/llvm/11/libc++/default.nix and pkgs/development/compilers/llvm/11/libc++abi.nix i see that there is some enableShared parametrization which hints at static binaries being possible, but from looking at the code i don't know if that is already supposed to work out of the box or still needs some tweaking.

Expected behavior

Using clang with libcxx on c++ programs with -static should result in successful compilation of static binaries.

Notify maintainers
@vlstill @lovek323 @dtzWill @primeos

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 5.9.16, NixOS, 20.09.2750.85abeab48b5 (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.10`
 - channels(root): `"nixos-20.09.2750.85abeab48b5, nixos-unstable-21.03pre261422.1a57d96edd1"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[ehmry]

@dtzWill Is there an LLVM version where this is known to work?

[sternenseemann]

pkgsStatic.libcxxStdenv (as opposed to pkgsStatic.llvmPackages.libcxxStdenv should work). Unfortunately the static adapters don't get applied to all llvm stdenvs atm.

#118313 should fix this for the most obvious cases (can you check your reproducer against that PR?).

[stale]

I marked this as stale due to inactivity. → More info

[pwaller]

cc @Ericson2314 @sternenseemann @Mic92 I think I've found a reason why static+llvm stdenvs are broken.

Another issue here is that;

1. The makeStaticBinaries stdenv (used to implement pkgsStatic) passes -static to the linker by setting NIX_CFLAGS_LINK:

nixpkgs/pkgs/stdenv/adapters.nix

Line 64 in b8960cd

NIX_CFLAGS_LINK = toString (finalAttrs.NIX_CFLAGS_LINK or "") + " -static";

2. GCC's compiler driver or the linker ignores -Wl,-dynamic-linker=... if you pass -static.

$ nix run nixpkgs#file $(nix build --print-out-paths nixpkgs#pkgsStatic.hello^out)/bin/hello
/nix/store/g6ay1bbrqcli3w51c9ghndw0j1f740ds-hello-static-x86_64-unknown-linux-musl-2.12.1/bin/hello:
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped

☝️ This one, built with the gcc stdenv, does not have a dynamic linker set.

2a) LLVM sets a dynamic linker anyway even if you pass -static:

$ nix build --keep-failed nixpkgs/0d8145a5d81ebf6698077b21042380a3a66a11c7#pkgsStatic.pkgsLLVM.hello

☝️ This fails during the check phase, because the produced executable segfault on startup.

👇 The reason they segfault is because the dynamic linker is running on the (static) binary, which does not work.

nix run nixpkgs#file /tmp/nix-build-hello-static-x86_64-unknown-linux-musl-*/hello-*/hello
/tmp/nix-build-hello-static-x86_64-unknown-linux-musl-2.12.1.drv-0/hello-2.12.1/hello: \
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, \
interpreter /nix/store/8lqwl0x6194hq7q862w8jn2wdxfb97fl-musl-static-x86_64-unknown-linux-musl-1.2.3/lib/ld-musl-x86_64.so.1, \
with debug_info, not stripped

3. -Wl,-dynamic-linker, inserted by cc-wrapper, is intended to be conditionalised on the $linkType.

nixpkgs/pkgs/build-support/cc-wrapper/cc-wrapper.sh

Lines 176 to 177 in b8960cd

	if [[ "$linkType" == dynamic && -n "$NIX_DYNAMIC_LINKER_@suffixSalt@" ]]; then
	extraBefore+=("-Wl,-dynamic-linker=$NIX_DYNAMIC_LINKER_@suffixSalt@")

4. The $linkType is inferred from the compiler wrappers arguments, which does not take $NIX_CFLAGS_LINK into account, resulting in $linkType == dynamic for pkgsStatic derivations.

nixpkgs/pkgs/build-support/wrapper-common/utils.bash

Lines 133 to 144 in a1bcfbc

	checkLinkType() {
	local arg
	type="dynamic"
	for arg in "$@"; do
	if [[ "$arg" = -static ]]; then
	type="static"
	elif [[ "$arg" = -static-pie ]]; then
	type="static-pie"
	fi
	done
	echo "$type"
	}

nixpkgs/pkgs/build-support/cc-wrapper/cc-wrapper.sh

Line 34 in b8960cd

linkType=$(checkLinkType "${params[@]}")

I think if this is fixed more things will work. I wonder if GCC and/or LLVM should refuse the combination of -Wl,-dynamic-linker=... -static in principle.

[Artturin]

@pwaller was this fixed by #253116 ?

[pwaller]

@pwaller was this fixed by #253116 ?

@Artturin I think it should be. I just tried to build the example but I hit #177129 in the build of libcxxabi-static-x86_64-unknown-linux-musl, during the configure step to check the compiler is working.

That is, that clang specifies gcc_eh to link, when it shouldn't, but this doesn't exist in a static-target gcc build. This is a clang issue, but clang doesn't have a way to know it shouldn't try to link gcc_eh. A workaround is to create an empty gcc_eh.

For libcxxabi, it could be worked around via CMAKE_C_COMPILER_WORKS, but I think a more general solution is needed. Perhaps to provide an empty gcc_eh in static builds so that clang's linker driver can function.

[pwaller]

There is an additional problem. -lc++abi is appearing on the link line before -lc++ which is the root cause of the original messages in this thread. Additionally, -lc needs to appear after both of these. With those changes, I get a binary I can run.
nix-support/libcxx-ldflags contains: -stdlib=libc++ -lc++abi. These are the flags clang++ is passing to the linker:

"-lc++abi"
"-lc++"
"-lm"
"--start-group"
"-lgcc"
"-lgcc_eh"
"-lc"
"--end-group"

So as a workaround, I can get a functioning binary by compiling the original example with $CXX $src -o hello -static -lc++.