nix: cross-compilation adds native bash to the closure

[flokli]

nix-build -A pkgsCross.armv7l-hf-multiplatform.nix and pkgsCross.aarch64-multiplatform.nix pull in the native bash, causing share/nix/corepkgs/config.nix to have shell point to an x86_64-linux executable:

[root@flokli:~/nixpkgs]# nix why-depends -a $(nix-build -A pkgsCross.aarch64-multiplatform.nix) $(nix-build -A bash)
/nix/store/k9zkv9fdch8n5v4c2x1c3p56j41f1kgl-nix-2.3.3-aarch64-unknown-linux-gnu
╠═══share/nix/corepkgs/config.nix: ….in rec {.  shell = "/nix/store/1iaxkm0941nj1m4m5g4fxgg4cq5jckf0-bash-4.4-p23/bin/bash";.  coreu…
║   => /nix/store/1iaxkm0941nj1m4m5g4fxgg4cq5jckf0-bash-4.4-p23
╠═══bin/nix: …nknown-linux-gnu/lib:/nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage…
║   lib/libboost_context.so.1.69.0: …nknown-linux-gnu/lib:/nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage…
║   lib/libboost_system.so.1.69.0: …nknown-linux-gnu/lib:/nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage…
║   lib/libnixexpr.so: …nknown-linux-gnu/lib:/nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage…
║   lib/libnixmain.so: …nknown-linux-gnu/lib:/nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage…
║   lib/libnixstore.so: …nknown-linux-gnu/lib:/nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage…
║   lib/libnixutil.so: …nknown-linux-gnu/lib:/nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage…
║   => /nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage-final-gcc-debug-9.2.0
║   ╚═══bin/aarch64-unknown-linux-gnu-c++: …lace-extension......./nix/store/mxhzy5v271gwz5ijl5w63444yb094lix-aarch64-unknown-linux-gnu-binut…
║       bin/aarch64-unknown-linux-gnu-cpp: …lace-extension......./nix/store/mxhzy5v271gwz5ijl5w63444yb094lix-aarch64-unknown-linux-gnu-binut…
║       bin/aarch64-unknown-linux-gnu-g++: …lace-extension......./nix/store/mxhzy5v271gwz5ijl5w63444yb094lix-aarch64-unknown-linux-gnu-binut…
║       bin/aarch64-unknown-linux-gnu-gcc: …lace-extension......./nix/store/mxhzy5v271gwz5ijl5w63444yb094lix-aarch64-unknown-linux-gnu-binut…
║       bin/aarch64-unknown-linux-gnu-gcc-9.2.0: …lace-extension......./nix/store/mxhzy5v271gwz5ijl5w63444yb094lix-aarch64-unknown-linux-gnu-binut…
║       lib/gcc/aarch64-unknown-linux-gnu/9.2.0/plugin/include/auto-host.h: …e DEFAULT_ASSEMBLER "/nix/store/mxhzy5v271gwz5ijl5w63444yb094lix-aarch64-unknown-linux-gnu-binut…
║       libexec/gcc/aarch64-unknown-linux-gnu/9.2.0/collect2: ….2.0/gcc/collect2.c../nix/store/mxhzy5v271gwz5ijl5w63444yb094lix-aarch64-unknown-linux-gnu-binut…
║       => /nix/store/mxhzy5v271gwz5ijl5w63444yb094lix-aarch64-unknown-linux-gnu-binutils-wrapper-2.31.1
║       ╚═══bin/aarch64-unknown-linux-gnu-ld: …#! /nix/store/1iaxkm0941nj1m4m5g4fxgg4cq5jckf0-bash-4.4-p23/bin/bash.set -eu -…
║           bin/aarch64-unknown-linux-gnu-ld.bfd: …#! /nix/store/1iaxkm0941nj1m4m5g4fxgg4cq5jckf0-bash-4.4-p23/bin/bash.set -eu -…
║           bin/aarch64-unknown-linux-gnu-ld.gold: …#! /nix/store/1iaxkm0941nj1m4m5g4fxgg4cq5jckf0-bash-4.4-p23/bin/bash.set -eu -…
║           => /nix/store/1iaxkm0941nj1m4m5g4fxgg4cq5jckf0-bash-4.4-p23
╠═══lib/libnixstore.so: …wn-linux-gnu-lib/lib:/nix/store/39in2k3lfv7g5i49fqpzr6rncdc7djl3-aws-sdk-cpp-1.7.90-aarch64-unkn…
║   => /nix/store/39in2k3lfv7g5i49fqpzr6rncdc7djl3-aws-sdk-cpp-1.7.90-aarch64-unknown-linux-gnu
║   ╚═══lib/libaws-cpp-sdk-core.so: …nknown-linux-gnu/lib:/nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage…
║       lib/libaws-cpp-sdk-s3.so: …nknown-linux-gnu/lib:/nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage…
║       lib/libaws-cpp-sdk-transfer.so: …nknown-linux-gnu/lib:/nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage…
║       => /nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage-final-gcc-debug-9.2.0
╚═══bin/nix: …nknown-linux-gnu/lib:/nix/store/wk348n7nz9ji28668qjawn8s1m652r1j-boehm-gc-8.0.4-aarch64-unknown-…
    lib/libnixexpr.so: …nknown-linux-gnu/lib:/nix/store/wk348n7nz9ji28668qjawn8s1m652r1j-boehm-gc-8.0.4-aarch64-unknown-…
    => /nix/store/wk348n7nz9ji28668qjawn8s1m652r1j-boehm-gc-8.0.4-aarch64-unknown-linux-gnu
    ╚═══lib/libgccpp.so.1.4.0: …nknown-linux-gnu/lib:/nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage…
        => /nix/store/x2xc85lxqcjw9d7nxy77xi6xy336k6vi-aarch64-unknown-linux-gnu-stage-final-gcc-debug-9.2.0

cc @samueldr @Ericson2314

[eamsden]

I wonder if the quick solution isn't a postUnpack hook that uses substitute --substForVar on corepkgs/config.nix.in to substitute in bash, coreutils, &c.

Further out, config.nix in nix HEAD no longer has paths in it, so this issue should go away. (It won't cross-compile to armv7l right now because rust throws a linker error when trying to build a cross-compile, but that's a separate issue).

[flokli]

yeah, this is just about the nix.src nixpkgs master currently points at - for the other things, we should probably open an issue in nixos/nix, if it isn't already - feel free to link to it.

[eamsden]

@flokli I don't know if an issue there would help as it's already been fixed in HEAD. It seems to have been fixed by the partial rust rewrite, though, so I'm not sure how easily the fix backports to the current (2.3) nix releases. Thus my suggestion of the temporary hack until cross-compiling works for what is currently the unstable version of nix (i.e. nixUnstable in nixpkgs)

[flokli]

Thus my suggestion of the temporary hack until cross-compiling works for what is currently the unstable version of nix (i.e. nixUnstable in nixpkgs)

Yeah, I think we should provide some working nix in nixpkgs. If the substitute --substForVar hack works, would you be up for adding a PR for it? This might also be useful to backport to 20.03 (cc @worldofpeace )

Further out, config.nix in nix HEAD no longer has paths in it, so this issue should go away. (It won't cross-compile to armv7l right now because rust throws a linker error when trying to build a cross-compile, but that's a separate issue).

nixUnstable also doesn't cross-compile for aarch64 due to said linker error. Is there an upstream bug tracking that?

[flokli]

This seems to be rustc failing to build, and somewhat reported in #68804.

[eamsden]

Yeah, I think we should provide some working nix in nixpkgs. If the substitute --substForVar hack works, would you be up for adding a PR for it? This might also be useful to backport to 20.03 (cc @worldofpeace )

I'll try to get to that later today. I have to look at how the paths are used in nix to see if it wants a path to the binary or just the base storepath.

[eamsden]

@flokli I think we can close this now since #81317 was merged.

The remaining build-platform deps of the runtime closure seem to be because the cross-compiling gcc puts its target (i.e. our host) platform binaries in its out output, so programs that are built with it and link against e.g. libstdc++ or libgcc_s dynamically are forced to depend on gcc and its closure. I think that should be a separate issue.

[flokli]

yes, this can be closed. Thanks for the fix!