glibc: can't cross compile

[elitak]

to reproduce:
nix-build pkgs/stdenv/linux/make-bootstrap-tools-cross.nix -A armv6l (edit: not armv5tel)

glibc fails because of the improper stripping done by pkgs/development/libraries/glibc/builder.sh at the end of postInstall(). This worked a while ago. Looks like it broke around 2016.02.28 maybe? There's another problem after the stripping one as well. If I get past it I'll maybe submit a PR, but I'm a bit afraid of breaking something crucial, what with this being glibc after all.

Incidentally, glibc's build files are pretty convoluted. I think the single most important library in nix/nixos should be a little better maintained, personally. By that I mean more comments, better organization of .nix and .sh, tests that catch these regressions, etc.

[dezgeg]

I have some working hacks in https://github.com/dezgeg/nixpkgs/tree/pr-aarch64 (for #13042). I will integrate them soon(TM), although I know they are already out of date and the recently merged closure-size branch breaks cross compilation in yet another ways...

[elitak]

Sorry, armv5tel is flat out broken for another reason. use -A armv6l to reproduce instead.

I'll try using your branches as a basis, I guess, but why are there no smoke tests for crossDrvs? I find super-basic mistakes like nativeBuildInputs not properly being separated from buildInputs all the time.

I plan to fix up some of these cross targets myself, with the end-goal being root FS generation for embedded systems. I'm not even close.

[dezgeg]

There is http://hydra.nixos.org/jobset/nixpkgs/cross-trunk but I don't think anyone has looked there in years.

I did build all the ARM tarballs (including armv5tel) quite recently, probably there are some useful commits left there as well:
cdef1cd

[elitak]

Wow, aggravating. Is there a reason why cross builds are neglected like this? It seems to me that the extra effort for keeping everything cross-buildable should be almost negligible--pretty much just maintaining (native/)buildInputs.

Should I bother trying to submit fixes to get that jobset a healthier shade of green?

I think I'll just use those tarballs and build natively, for now.

[vcunat]

Is there a reason why cross builds are neglected like this?

As usual, it's just noone really working on them around nixos.org AFAIK. I'll be shortly proposing/pushing some changes around linux->mingw cross-building, but I suppose that's rather different from what you're interested in...

[elitak]

Where can I get the .sql or whatever to import this cross-trunk jobset into my own Hydra server?

[elitak]

Things have further deteriorated on nixpkgs master:

~/nixpkgs$ nix-build pkgs/stdenv/linux/make-bootstrap-tools-cross.nix -A armv7l  --show-trace
error: while evaluating the attribute ‘buildCommand’ of the derivation ‘build’ at pkgs/stdenv/linux/make-bootstrap-tools-cross.nix:122:7:
while evaluating the attribute ‘buildInputs’ of the derivation ‘glibc-2.23-armv7l-unknown-linux-gnueabi’ at pkgs/development/libraries/glibc/common.nix:139:3:
while evaluating the attribute ‘gcc’ of the derivation ‘gcc-cross-wrapper’ at pkgs/build-support/gcc-cross-wrapper/default.nix:51:3:
while evaluating the attribute ‘nativeBuildInputs’ of the derivation ‘gcc-5.3.0-armv7l-unknown-linux-gnueabi-stage-static’ at pkgs/development/compilers/gcc/5/default.nix:209:3:
while evaluating the attribute ‘nativeBuildInputs’ of the derivation ‘gettext-0.19.7’ at pkgs/development/libraries/gettext/default.nix:4:3:
infinite recursion encountered, at undefined position

Something to do with the change to the import line, probably for the closure-size patches. I'm at a real loss here. I'd like to improve this build script, but I'm blocked.

[dezgeg]

Seems like it's just a bad merge resolution from the latest commit that touched the file:

-   buildInputs = [ xz xz.bin ] ++ stdenv.lib.optional (!stdenv.isLinux) libiconv;
 -  buildInputs = [ xz ] ++ lib.optional (!stdenv.isLinux) libiconv;
++  buildInputs = [ xz xz.bin libiconv ];

[dezgeg]

Where can I get the .sql or whatever to import this cross-trunk jobset into my own Hydra server?

I don't know anything about actually configuring Hydra but the Configuration tab tells it's pkgs/top-level/release-cross.nix

[vcunat]

The jobset config is just a few fields, so it's quick enough to "copy by hand". I don't know about any other way.

[vcunat]

On linux we should have libiconv = glibc. The thing is that stdenv.isLinux is currently a wrong condition if cross-compiling from linux to non-linux, though I'd hope to fix that around #10874. EDIT: of course, bootstrapping for some platforms may not count on this.

[elitak]

When I remove the libiconv input in gettext and fix the bad stripping in glibc's builder.sh, I get:

shrinking /nix/store/hp8b3n8d8rf8prywdzrdz3jdzqpc6s3q-glibc-2.23-armv7l-unknown-linux-gnueabi-static/lib/libieee.a
wrong ELF type
patching script interpreter paths in /nix/store/hp8b3n8d8rf8prywdzrdz3jdzqpc6s3q-glibc-2.23-armv7l-unknown-linux-gnueabi-static
shrinking RPATHs of ELF executables and libraries in /nix/store/l00n1xxqkrkwwsil68c0cf0jrgggn7vx-glibc-2.23-armv7l-unknown-linux-gnueabi-debug
find: '/nix/store/l00n1xxqkrkwwsil68c0cf0jrgggn7vx-glibc-2.23-armv7l-unknown-linux-gnueabi-debug': No such file or directory
crossStrip
cross stripping (with flags -S) in  /nix/store/66ssykk2h76lfhr3mzksj7qvwyjp52lq-glibc-2.23-armv7l-unknown-linux-gnueabi/lib  /nix/store/66ssykk2h76lfhr3mzksj7qvwyjp52lq-glibc-2.23-armv7l-unknown-linux-gnueabi/lib64  /nix/store/66ssykk2h76lfhr3mzksj7qvwyjp52lq-glibc-2.23-armv7l-unknown-linux-gnueabi/libexec
armv7l-unknown-linux-gnueabi-strip:/nix/store/66ssykk2h76lfhr3mzksj7qvwyjp52lq-glibc-2.23-armv7l-unknown-linux-gnueabi/lib/libc.so: File format not recognized
armv7l-unknown-linux-gnueabi-strip:/nix/store/66ssykk2h76lfhr3mzksj7qvwyjp52lq-glibc-2.23-armv7l-unknown-linux-gnueabi/lib/libpthread.so: File format not recognized
armv7l-unknown-linux-gnueabi-strip:/nix/store/66ssykk2h76lfhr3mzksj7qvwyjp52lq-glibc-2.23-armv7l-unknown-linux-gnueabi/lib/gconv/gconv-modules: File format not recognized
builder for ‘/nix/store/0bm6vx6x2mr6smrx2q120m6vx0h4l9jv-glibc-2.23-armv7l-unknown-linux-gnueabi.drv’ failed to produce output path ‘/nix/store/l00n1xxqkrkwwsil68c0cf0jrgggn7vx-glibc-2.23-armv7l-unknown-linux-gnueabi-debug’
cannot build derivation ‘/nix/store/0w447vsfk7wjkcr3qwyp2hkmj08vqnsh-build.drv’: 1 dependencies couldn't be built

Why's the build expecting -static and -debug suffixed output paths? is this closure-size related?

[vcunat]

We've changed the way we build glibc by default, and things just aren't that simple to work in every platform/config.

The "static" output is due to outputs = [ ... "static" ] and "debug" output due to separateDebugInfo = true;.

[elitak]

Okay, thanks everyone for the hints; I will try to use them to progress on my own.

[vcunat]

The fastest work around this problem should be to just mkdir -p "$static"; mkdir -p "$debug".

[elitak]

separateDebugInfo = true is within the crossDrv, but not nativeDrv. This is one of the few places in the entire nixpkgs tree where the feature is used, and it doesn't work, at least for the arch I'm targeting. I'm confused as to why it's even there. What good is it to produce an empty tree? I'll try to debug further; I just need to rebuild the whole thing again since I lost the output.

Incidentally, is is possible to complete building a derivation that's failed, but preserved with --keep-failed? §14.4.1 of the manual doesn't show any way to complete the build, only debug the phases individually. Basically, I want the Nix equivalent of ebuild install qmerge from Gentoo.

[vcunat]

On normal compilation that feature works well. For now it should be easiest for you to e.g. remove that separateDebugInfo flag or some work-around like suggested above, until someone fixes it with cross-building.

When a build fails, typically there's little sense in "forcing it to succeed", so I don't think there's any command for that.

[elitak]

nixpkgs/pkgs/development/compilers/gcc/builder.sh

Line 74 in 90ee003

EXTRA_TARGET_CFLAGS="-B${libcCross}/lib -idirafter ${libcCross}/include"

Here, the builder tries to use a single libc derivation for both /lib and /include. By default, this is set to the "dev" output, but there is no single output that provides both. Is the best way to address this to add and pass in libcCross_dev to point to libCross.dev, leaving libCross pointing to libCross.out? (

I also don't understand where/why libcCross.dev is being passed into the gcc-cross-final derivation rather than libcCross.out. I had to force it back to "out" like this:

64876a1#diff-036410e9211b4336186fc613f7200b12L6870

( appending .out to glibcCross value in all-packages.nix ). What is the proper way to address this?

[elitak]

Even after all those fixes, when building gcc--stage-final, I get:

/nix/store/5ns97xnqdscj0azkdsd4i7bpjp3k3kgr-binutils-2.26-armv7a-hardfloat-linux-gnueabi/bin/armv7a-hardfloat-linux-gnueabi-ld: skipping incompatible /nix/store/nfy3fz9q5zsppqp4kxdmmcw3aaq4raxc-glibc-2.23-armv7a-hardfloat-linux-gnueabi/lib/libc_nonshared.a when searching for /nix/store/nfy3fz9q5zsppqp4kxdmmcw3aaq4raxc-glibc-2.23-armv7a-hardfloat-linux-gnueabi/lib/libc_nonshared.a
/nix/store/5ns97xnqdscj0azkdsd4i7bpjp3k3kgr-binutils-2.26-armv7a-hardfloat-linux-gnueabi/bin/armv7a-hardfloat-linux-gnueabi-ld: cannot find /nix/store/nfy3fz9q5zsppqp4kxdmmcw3aaq4raxc-glibc-2.23-armv7a-hardfloat-linux-gnueabi/lib/libc_nonshared.a
collect2: error: ld returned 1 exit status
Makefile:947: recipe for target 'libgcc_s.so' failed

This error comes after the ultimate link step that is huge (31k chars), ./gcc/xgcc. The problem library is in the ${libcCross.out} path passed to the linker via -Wl,-L. How is this cross-target libc path not appropriate for gcc-cross? I really don't get what I'm missing.

[elitak]

I've finally gotten working cross builds on the master branch. The weirdest bit was that something was breaking libc_nonshared.a and libpthread_nonshared.a in the glibc.crossDrv install or fixup phase, rewriting the object headers of a single object file in each, resulting in "incompatible object type" errors later down the line when trying to link against the glibc.crossDrv. I'm still not sure what exact line was doing this. Right now I've just got a hack in place to copy over these libs again in postFixup. Once I figure out what exactly is the problem, I'll submit a PR to get at least armv5/7 cross building working again. Mingw32/64 still have issues rooted in the closure-size changes.

Specifically, the machine type given by file elf-init.oS (extracted from libc_nonshared.a) was returning ARM before the installPhase, then no machine after it. The other .oS in the lib are unaffected. Does anyone know what could be the cause of this? I checked the commit histories and see to obvious candidates for a regression; all I know is that this was not a problem pre-closure-size merge.

[elitak]

Looks like it was just the native strip being used explicitly in postInstall when the automatic crossStrip was going to run anyway. I'm still confused about how/why strip does this, but whatever. I'll collect everything into commits for a PR.

[elitak]

Pull request #15867 solves all the issues I was having.