llvmPackages_15: Building with clang++ using libcxx/-stdlib=libc++ fails

[LogicalOverflow]

Describe the bug

Using -lc++ with clang++ seems to be broken with llvmPackages_15.clang and llvmPackages_15.libcxxClang. It fails with ld: cannot find l: No such file or directory. #213144 also notes this.

Steps To Reproduce

Running clang++ -stdlib=libc++ main.cpp yields -/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: cannot find l: No such file or directory. The main.cpp contains just an empty main() and I am using this flake's devShell:

{
  description = "Test";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }: let
    lib = nixpkgs.lib;
    pkgs = nixpkgs.legacyPackages.x86_64-linux;
    llvmPkgs = pkgs.llvmPackages_15;
  in {
    devShells.x86_64-linux.default = pkgs.mkShell {
      packages = [
        llvmPkgs.clang

        llvmPkgs.libcxx
        llvmPkgs.libcxxabi
      ];
    };
  };
}

The same also occurs when using a shell with only llvmPkgs.libcxxClang and when trying to use libcxx inside a nix build.

I am pretty sure that this is caused by lib/libc++.so in llvmPackages_15.libcxx containing INPUT(libc++.so.1 -l) instead of INPUT(libc++.so.1 -lc++abi) (which it does for llvmPackages_14.libcxx). I don't know yet, what the root cause for that is or how to fix that.

Notify maintainers

@rrbutani @Ericson2314 @sternenseemann @lovek323 @dtzWill @primeos

(So it shows up in the tracking issue #213033)

[rrbutani]

Apologies for the delay in getting to this.

Minimizing your example ultimately leaves me with:

/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld \
    -L/nix/store/a1z9c4pmipjzcimqjzr8hlc3m143acwi-libcxx-15.0.7/lib \
    '-lc++'

Staring at the output of ^ with --verbose reveals that libc++.so is a linker script that appears to be passing in -l:

GNU ld (GNU Binutils) 2.39
  Supported emulations:
   elf_x86_64
   elf32_x86_64
   elf_i386
   elf_iamcu
using internal linker script:
<snipped>
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: mode elf_x86_64
attempt to open /nix/store/a1z9c4pmipjzcimqjzr8hlc3m143acwi-libcxx-15.0.7/lib/libc++.so succeeded
opened script file /nix/store/a1z9c4pmipjzcimqjzr8hlc3m143acwi-libcxx-15.0.7/lib/libc++.so
/nix/store/a1z9c4pmipjzcimqjzr8hlc3m143acwi-libcxx-15.0.7/lib/libc++.so
opened script file /nix/store/a1z9c4pmipjzcimqjzr8hlc3m143acwi-libcxx-15.0.7/lib/libc++.so
-attempt to open /nix/store/a1z9c4pmipjzcimqjzr8hlc3m143acwi-libcxx-15.0.7/lib/libc++.so.1 succeeded
/nix/store/a1z9c4pmipjzcimqjzr8hlc3m143acwi-libcxx-15.0.7/lib/libc++.so.1
attempt to open /nix/store/a1z9c4pmipjzcimqjzr8hlc3m143acwi-libcxx-15.0.7/lib/l failed
attempt to open l failed
attempt to open /nix/store/a1z9c4pmipjzcimqjzr8hlc3m143acwi-libcxx-15.0.7/lib/l failed

For LLVM 14 this file looks like:

$ nix-shell --expr "with (import <nixpkgs> {}); mkShell { libcxx = llvmPackages_14.libcxx; }" --run 'cat $libcxx/lib/libc++.so'
INPUT(libc++.so.1 -lc++abi)

But for LLVM 15:

$ nix-shell --expr "with (import <nixpkgs> {}); mkShell { libcxx = llvmPackages_15.libcxx; }" --run 'cat $libcxx/lib/libc++.so'
INPUT(libc++.so.1 -l)

EDIT: whoops, my bad; just noticed that you already mentioned this in your post, I totally missed it

---

Looking through the libcxx source, this file appears to be generated here; specifically the -lc++abi should be coming from here based on libcxx-abi-shared. For Linux/macOS (i.e. platforms using libc++-abi) that should be set here.

Poking around a bit suggests that this is a bug in upstream; libcxx-abi-shared is not an interface library so it has no IMPORTED_LIBNAME (though it has an IMPORTED_LOCATION) and its OUTPUT_NAME is never set.

And sure enough, looking at upstream reveals that has been patched in LLVM16+ (see here).

[rrbutani]

Applying that patch and rebuilding libcxx locally seems to fix the issue; I'll open a PR (and add regression tests for this) shortly.

[LogicalOverflow]

Thanks for looking into this! Building from the PR-branch now no longer fails in the way I described and the example works for me again.

I have now found a new issue, tho: Importing cxxlib.h fails with:

/nix/store/qwnvng0cbyx0bijm654jpmpl0516hfhx-libcxxabi-15.0.7-dev/include/cxxabi.h:20:10: fatal error: '__cxxabi_config.h' file not found

For example, this main.cpp causes this error (using the same flake as above, just built from the PR branch).

#include<cxxabi.h>
int main() {}

I was able to work around this, by adding

install -m 644 ../../${pname}/include/__cxxabi_config.h "$dev/include"

to the end of the postInstall phase of libcxxabi locally, but I am not sure if that's the right approach here.

[pwaller]

(Re @LogicalOverflow) I have a proposed fix for the __cxxabi_config.h issue on llvmPackages_16: #246577 -- currently I'm inserting an additional include search dir for it. I also don't know if this is the best approach or alternatively if that header (and friends?) should be copied into libcxx or similar.

[magneticflux-]

I ran into the same missing header issue compiling c-ares-1.19.1-i686-linux using pkgs.llvmPackages_16.libcxxStdenv on nixos-unstable-small. Maybe the fix isn't enough, and __cxxabi_config.h should be copied? 6ba1b5b (#246577)

[Artturin]

Thanks for looking into this! Building from the PR-branch now no longer fails in the way I described and the example works for me again.

I have now found a new issue, tho: Importing cxxlib.h fails with:

/nix/store/qwnvng0cbyx0bijm654jpmpl0516hfhx-libcxxabi-15.0.7-dev/include/cxxabi.h:20:10: fatal error: '__cxxabi_config.h' file not found

For example, this main.cpp causes this error (using the same flake as above, just built from the PR branch).

#include<cxxabi.h>
int main() {}

I was able to work around this, by adding

install -m 644 ../../${pname}/include/__cxxabi_config.h "$dev/include"

to the end of the postInstall phase of libcxxabi locally, but I am not sure if that's the right approach here.

this change is missing from 15 6ba1b5b#diff-10d812ae0081a7211c31f286a68130450af9ff7efa30fff60a337022816df01dR264

[natto1784]

running into the same problem compiling some static libraries in nixpkgs using pkgs.llvmPackages_16.libcxxStdenv. Does this warrant its own issue?

[pwaller]

@natto1784 Yes, file a new issue. Please also provide reproducibility bits, including the error message you see, the package expression you're trying to build, and the nixpkgs revision. I'm guessing the issue you're seeing is pkgsStatic.llvmPackages_16.libcxxabi failing to build, which is a different problem specific to static builds.

[pwaller]

@natto1784 (If my guess is accurate, the problem you're seeing is a variant of #177129).

[natto1784]

@pwaller sorry for bothering you, turns out it wasn't really this issue. Using my limited knowledge on the matter from what I understand, $src/libcxxabi/include/__cxxabi_config.h is not installed to $dev/include like $src/libcxxabi/include/cxxabi.h.

Now, turns out ${libcxx.dev}/include/c++/v1 (and ${libcxx.dev}/include) is a default system header path (for CXX of course) and so is ${libcxxabi.dev}/include BUT ${libcxxabi.dev}/include/c++/v1 is not so if one were to #include <cxxabi.h> they would just run into #include <__cxxabi_config.h> not found since that is nowhere to be found.

There are two solutions to this

1. Either ${libcxxabi.dev}/include/c++/v1 is added as a system header path like ${libcxx.dev}/include/c++/v1
2. Or we simply just install everything under $src/libcxxabi/include on non darwin systems as well. I actually do not know why these are all not copied on non darwin systems.

Please correct me if wrong. I might be completely wrong.

[Artturin]

@pwaller sorry for bothering you, turns out it wasn't really this issue. Using my limited knowledge on the matter from what I understand, $src/libcxxabi/include/__cxxabi_config.h is not installed to $dev/include like $src/libcxxabi/include/cxxabi.h.

Now, turns out ${libcxx.dev}/include/c++/v1 (and ${libcxx.dev}/include) is a default system header path (for CXX of course) and so is ${libcxxabi.dev}/include BUT ${libcxxabi.dev}/include/c++/v1 is not so if one were to #include <cxxabi.h> they would just run into #include <__cxxabi_config.h> not found since that is nowhere to be found.

There are two solutions to this

1. Either `${libcxxabi.dev}/include/c++/v1` is added as a system header path like `${libcxx.dev}/include/c++/v1`

2. Or we simply just install everything under `$src/libcxxabi/include` on non darwin systems as well. I actually do not know why these are all not copied on non darwin systems.

Please correct me if wrong. I might be completely wrong.

check #216273

[pwaller]

@pwaller sorry for bothering you

No bother. I'm sticking my oar in because I'm interested in getting to the bottom of issues exactly like this. It's difficult to fix things it without precise reproduction steps, though.

What you're describing there sounds very much like something I tried to address in #246577 (see 6ba1b5b#diff-10d812ae0081a7211c31f286a68130450af9ff7efa30fff60a337022816df01dR257-R264), which is on nixos-unstable. So it sounds like you may have found further issues, but I can't do much to help without a reproducer.

In any case what you're describing sounds like a different issue to the original (which was ld: cannot find l: No such file or directory), so I suggest posting a new issue pretty much as you described but with a reproducer I can run at the terminal, since it is meant to be in the system include path already (per the link above). If it's not, that sounds like a bug worth filing.

[natto1784]

Okay I found the problem, and the "obvious" solution as well, you can find it here #255487 @pwaller