python: don't use keep for hooks, so they will be spliced

[Artturin]

See #211340 for prerequisite

wrapPython for some reason has to be not spliced because 'substitutions.pythonHost = python;' causes

error: output '/nix/store/li07bad6s9brxx7cf8j408631nca39ka-python3.10-chardet-5.1.0-aarch64-unknown-linux-gnu' is not allowed to refer to the following paths:
         /nix/store/fdqpyj613dr0v1l1lrzqhzay7sk4xg87-python3-3.10.10

ericson said in #104201 (comment) that the reason they were in keep was to not splice them but before the commit which broke hook splicing (33d12e5) everything appeared to be working correctly when cross-compiling

Description of changes

experimenting with hook splicing fix @amjoseph-nixpkgs

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[Artturin]

@Ericson2314 maybe made them not splice to work around #211340

nixpkgs/pkgs/development/tools/misc/itstool/default.nix

Line 20 in 7482a0e

python3.pkgs.wrapPython

$ nix build ".#pkgsCross.aarch64-multiplatform.itstool"
error:
       … while calling the 'derivationStrict' builtin

         at <nix/derivation-internal.nix>:9:12:

            8|
            9|   strict = derivationStrict drvAttrs;
             |            ^
           10|

       … while evaluating derivation 'itstool-aarch64-unknown-linux-gnu-2.0.6'
         whose name attribute is located at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:300:7

       … while evaluating attribute 'nativeBuildInputs' of derivation 'itstool-aarch64-unknown-linux-gnu-2.0.6'

         at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:344:7:

          343|       depsBuildBuild              = lib.elemAt (lib.elemAt dependencies 0) 0;
          344|       nativeBuildInputs           = lib.elemAt (lib.elemAt dependencies 0) 1;
             |       ^
          345|       depsBuildTarget             = lib.elemAt (lib.elemAt dependencies 0) 2;

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: makeWrapper/makeShellWrapper must be in nativeBuildInputs

[ghost]

Let me be totally honest here, I'm trying to remove setup hooks from nixpkgs -- starting from the cc-wrapper and binutils-wrapper which is where this whole mess begins. #238069 is the first steps towards that.

The main reason why setup hooks exist is that there are pieces of data that get stuffed into $out/nix-support instead of in Nix attrsets. Once that happens, you can't branch/select/switch on those values at eval time (without IFD of course). When these values are used to compute other values, you get an "avalanche" of logic that gets forced out of the Nix eval language and into hideous gross piles of bash.

It turns out that 99% of this stuff is actually knowable at Eval time. But you have to stop the avalanche before it starts.

That is my mission.

With all that said, @Artturin I appreciate your heroic efforts to tame the bash-monster, and we need these efforts. I just wanted to explain why it's really hard for me to motivate myself to get a deep enough understanding of problems with this stuff on one hand when I'm putting a lot of effort into being able to get rid of it on the other.

It's also starting to look like I'm probably the only even-semi-qualified reviewer for a lot of this stuff. I will give it a try. Please keep pestering me.

I'm going to cherry-pick this into my local tree and rebuild my 100%-cross platforms (aarch64 and mips) overnight to see what breaks.

[FRidh]

It turns out that 99% of this stuff is actually knowable at Eval time. But you have to stop the avalanche before it starts.

This is interesting. I've been wanting this as well, however, a big worry is that eval time increases.

E.g., with Python we still use propagated-build-inputs. We can get rid of that in two ways, either writing a custom file linking runtime deps, or keeping track of deps in eval only and writing references out when building an env.

[ghost]

Addendum: the kicker starts with this commit, which isn't part of the PR I linked (but goes immediately on top of it). This commit does not eval, but it's sort of the first milestone I've been working towards:

• https://github.com/amjoseph-nixpkgs/nixpkgs/commit/3d45166c6e46d8a3732532745b07da4d97029bc4

keeping track of deps in eval only and writing references out when building an env

You definitely want to do this.

I've been wanting this as well, however, a big worry is that eval time increases.

Why would it though? The evaluator still has to instantiate all ancestral dependencies either way. So the number of derivations instantiated is the same either way; the only difference is whether the list is passed as lines in nix-support/propagated-build-inputs or as a Nix list.

It might make sense to "keep track of deps" using an attrset rather than a list though, since unioning n-many one-element attrsets is O(nlog n) (cppnix implements this by merging pre-sorted lists) but concatenating n-many one-element lists is O(n^2). Nevermind. In the truly, truly pathological case they're both O(n^2). However we can use the fact that attrsets are implemented as sorted lists (with binary search) to build better data structures; it just isn't automatic.

[ghost]

This breaks eval for claws-mail (which I haven't actually used in over a year) but everything else was fine. Builds underway; I'll check back in the morning.

Built on:

• aarch64-linux laptop packageset (cross from x86_64-linux)
• mips64el-linux router packageset (cross from x86_64-linux)
  • abi=64 (stuff that uses rust, golang, and boost-context)
  • abi=n32 (everything else)

Edit: got up to firefox, chromium and qutebrowser (all three still building but I expect they will succeed) with only four failures. I'll diagnose those in the morning.

[ghost]

Hrm, so my biggest concern is that #227902 no longer works in the presence of this PR. Also (and independently), qtwebengine no longer cross-compiles.

I'm digging to find out why.

Other fails are:

python-frozenlist

error: builder for '/nix/store/ci6hmcv0niclbd3vqrhr1zg19x9gq25b-python3.10-frozenlist-1.3.3-aarch64-unknown-linux-gnu.drv' failed with exit code 1;
       last 10 log lines:
       > patching sources
       > updateAutotoolsGnuConfigScriptsPhase
       > updateAutotoolsGnuConfigScriptsPhase
       > configuring
       > no configure script, doing nothing
       > building
       > Executing setuptoolsBuildPhase
       > /nix/store/g8m639c6973656mlg2xaiqh3b88grkbs-python3.10-cython-0.29.34/bin/.cython-wrapped: /nix/store/g8m639c6973656mlg2xaiqh3b88grkbs-python3.10-cython-0.29.34/bin/cython: line 3: syntax error near unexpected token `lambda'
       > /nix/store/g8m639c6973656mlg2xaiqh3b88grkbs-python3.10-cython-0.29.34/bin/.cython-wrapped: /nix/store/g8m639c6973656mlg2xaiqh3b88grkbs-python3.10-cython-0.29.34/bin/cython: line 3: `import sys;import site;import functools;sys.argv[0] = '/nix/store/g8m639c6973656mlg2xaiqh3b88grkbs-python3.10-cython-0.29.34/bin/cython';functools.reduce(lambda k, p: site.addsitedir(p, k), ['/nix/store/g8m639c6973656mlg2xaiqh3b88grkbs-python3.10-cython-0.29.34/lib/python3.10/site-packages','/nix/store/ci5b14j87px5h2j83244qrdxzy9c89lc-python3-3.10.12/lib/python3.10/site-packages'], site._init_pathinfo());'
       > /nix/store/giycz6b15x39q4ffaabcjaimv3p57pis-stdenv-linux/setup: line 144: pop_var_context: head of shell_variables not a function context

aiosignal

error: 1 dependencies of derivation '/nix/store/h0s4w0dymgg3sq2qymk9xk1qfjdalj0h-python3.10-aiosignal-1.3.1-aarch64-unknown-linux-gnu.drv' failed to build
error: build of '/nix/store/wxibxw134xjkgrqxyak7llqialpm8m5a-python3.10-webencodings-0.5.1.drv' on 'ssh://[email protected]' failed: builder for '/nix/store/wxibxw134xjkgrqxyak7llqialpm8m5a-python3.10-webencodings-0.5.1.drv' failed with exit code 2
error: builder for '/nix/store/wxibxw134xjkgrqxyak7llqialpm8m5a-python3.10-webencodings-0.5.1.drv' failed with exit code 1;
       last 10 log lines:
       > patching script interpreter paths in /nix/store/7158a4001i2spia03v71jx88nkaavnkl-python3.10-webencodings-0.5.1
       > stripping (with command strip and flags -S -p) in  /nix/store/7158a4001i2spia03v71jx88nkaavnkl-python3.10-webencodings-0.5.1/lib
       > shrinking RPATHs of ELF executables and libraries in /nix/store/4qnpss02qlvap8ifwhk0s90nnn8y3s2l-python3.10-webencodings-0.5.1-dist
       > checking for references to /build/ in /nix/store/4qnpss02qlvap8ifwhk0s90nnn8y3s2l-python3.10-webencodings-0.5.1-dist...
       > patching script interpreter paths in /nix/store/4qnpss02qlvap8ifwhk0s90nnn8y3s2l-python3.10-webencodings-0.5.1-dist
       > Executing pythonRemoveTestsDir
       > Finished executing pythonRemoveTestsDir
       > running install tests
       > /nix/store/nlhs8d6nrdzk4kqv173gzj0igb0j7ikk-python3.10-pytest-7.2.1/bin/.py.test-wrapped: /nix/store/nlhs8d6nrdzk4kqv173gzj0igb0j7ikk-python3.10-pytest-7.2.1/bin/py.test: line 3: syntax error near unexpected token `lambda'
       > /nix/store/nlhs8d6nrdzk4kqv173gzj0igb0j7ikk-python3.10-pytest-7.2.1/bin/.py.test-wrapped: /nix/store/nlhs8d6nrdzk4kqv173gzj0igb0j7ikk-python3.10-pytest-7.2.1/bin/py.test: line 3: `import sys;import site;import functools;sys.argv[0] = '/nix/store/nlhs8d6nrdzk4kqv173gzj0igb0j7ikk-python3.10-pytest-7.2.1/bin/py.test';functools.reduce(lambda k, p: site.addsitedir(p, k), ['/nix/store/nlhs8d6nrdzk4kqv173gzj0igb0j7ikk-python3.10-pytest-7.2.1/lib/python3.10/site-packages','/nix/store/fb1wnff0hgr0spxg3nrqyyrav748djcy-python3.10-attrs-22.2.0/lib/python3.10/site-packages','/nix/store/ci5b14j87px5h2j83244qrdxzy9c89lc-python3-3.10.12/lib/python3.10/site-packages','/nix/store/fmh8m17gz5sv22924wzyscmq6sjnvi0m-python3.10-iniconfig-2.0.0/lib/python3.10/site-packages','/nix/store/gfikmbv231cw767abgwym6bilpw2zqh4-python3.10-packaging-23.0/lib/python3.10/site-packages','/nix/store/yswzkw3s0qrycx66hr9265m4m6z8fja2-python3.10-pluggy-1.0.0/lib/python3.10/site-packages','/nix/store/yab91szh63mkj5mdf6xzp2x95fdl8sz8-python3.10-py-1.11.0/lib/python3.10/site-packages','/nix/store/j46b4k3z06djd451x4arkwy6b408vpyf-python3.10-tomli-2.0.1/lib/python3.10/site-packages','/nix/store/4q783kasfnf7b22rlhka87bjqqghwaxs-python3.10-exceptiongroup-1.1.0/lib/python3.10/site-packages'], site._init_pathinfo());'

python-flaky

error: build of '/nix/store/h3k4r2dbpb7jvl37b2yndv9jsc4agdam-python3.10-flaky-3.7.0.drv' on 'ssh://[email protected]' failed: builder for '/nix/store/h3k4r2dbpb7jvl37b2yndv9jsc4agdam-python3.10-flaky-3.7.0.drv' failed with exit code 2
error: builder for '/nix/store/h3k4r2dbpb7jvl37b2yndv9jsc4agdam-python3.10-flaky-3.7.0.drv' failed with exit code 1;
       last 10 log lines:
       > patching script interpreter paths in /nix/store/gz768x9yvp10n8lk4hd50zggz9dybi7h-python3.10-flaky-3.7.0
       > stripping (with command strip and flags -S -p) in  /nix/store/gz768x9yvp10n8lk4hd50zggz9dybi7h-python3.10-flaky-3.7.0/lib
       > shrinking RPATHs of ELF executables and libraries in /nix/store/jlqmcqkr120m45hd2q9f8kwnvkpjl8ay-python3.10-flaky-3.7.0-dist
       > checking for references to /build/ in /nix/store/jlqmcqkr120m45hd2q9f8kwnvkpjl8ay-python3.10-flaky-3.7.0-dist...
       > patching script interpreter paths in /nix/store/jlqmcqkr120m45hd2q9f8kwnvkpjl8ay-python3.10-flaky-3.7.0-dist
       > Executing pythonRemoveTestsDir
       > Finished executing pythonRemoveTestsDir
       > running install tests
       > /nix/store/nlhs8d6nrdzk4kqv173gzj0igb0j7ikk-python3.10-pytest-7.2.1/bin/.pytest-wrapped: /nix/store/nlhs8d6nrdzk4kqv173gzj0igb0j7ikk-python3.10-pytest-7.2.1/bin/pytest: line 3: syntax error near unexpected token `lambda'
       > /nix/store/nlhs8d6nrdzk4kqv173gzj0igb0j7ikk-python3.10-pytest-7.2.1/bin/.pytest-wrapped: /nix/store/nlhs8d6nrdzk4kqv173gzj0igb0j7ikk-python3.10-pytest-7.2.1/bin/pytest: line 3: `import sys;import site;import functools;sys.argv[0] = '/nix/store/nlhs8d6nrdzk4kqv173gzj0igb0j7ikk-python3.10-pytest-7.2.1/bin/pytest';functools.reduce(lambda k, p: site.addsitedir(p, k), ['/nix/store/nlhs8d6nrdzk4kqv173gzj0igb0j7ikk-python3.10-pytest-7.2.1/lib/python3.10/site-packages','/nix/store/fb1wnff0hgr0spxg3nrqyyrav748djcy-python3.10-attrs-22.2.0/lib/python3.10/site-packages','/nix/store/ci5b14j87px5h2j83244qrdxzy9c89lc-python3-3.10.12/lib/python3.10/site-packages','/nix/store/fmh8m17gz5sv22924wzyscmq6sjnvi0m-python3.10-iniconfig-2.0.0/lib/python3.10/site-packages','/nix/store/gfikmbv231cw767abgwym6bilpw2zqh4-python3.10-packaging-23.0/lib/python3.10/site-packages','/nix/store/yswzkw3s0qrycx66hr9265m4m6z8fja2-python3.10-pluggy-1.0.0/lib/python3.10/site-packages','/nix/store/yab91szh63mkj5mdf6xzp2x95fdl8sz8-python3.10-py-1.11.0/lib/python3.10/site-packages','/nix/store/j46b4k3z06djd451x4arkwy6b408vpyf-python3.10-tomli-2.0.1/lib/python3.10/site-packages','/nix/store/4q783kasfnf7b22rlhka87bjqqghwaxs-python3.10-exceptiongroup-1.1.0/lib/python3.10/site-packages'], site._init_pathinfo());'
       For full logs, run 'nix log /nix/store/h3k4r2dbpb7jvl37b2yndv9jsc4agdam-python3.10-flaky-3.7.0.drv'.

python3-cryptography

error: builder for '/nix/store/k9wrb5z6gnj5xiymdjs6pzwlkq746ss7-python3.10-cryptography-40.0.1-aarch64-unknown-linux-gnu.drv' failed with exit code 1;
       last 10 log lines:
       >     Python: 3.10.12
       >     platform: Linux-5.15.82-x86_64-with-glibc2.37
       >     pip: 23.0.1
       >     setuptools: 67.4.0.post0
       >     setuptools_rust: 1.5.2
       >     rustc: 1.70.0 (90c541806 2023-05-31) (built from a source tarball)
       >     =============================DEBUG ASSISTANCE=============================
       >
       > error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path src/rust/Cargo.toml --target x86_64-unknown-linux-gnu --release -v --features 'pyo3/abi3-py36 pyo3/extension-module' -- --crate-type cdylib` failed with code 101
       > /nix/store/giycz6b15x39q4ffaabcjaimv3p57pis-stdenv-linux/setup: line 1596: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/k9wrb5z6gnj5xiymdjs6pzwlkq746ss7-python3.10-cryptography-40.0.1-aarch64-unknown-linux-gnu.drv'.

[Artturin]

useful
nix-diff $(nix eval --raw "nixpkgs/$(git merge-base staging HEAD)#pkgsCross.aarch64-multiplatform.python3Packages.frozenlist.drvPath") $(nix eval --raw ".#pkgsCross.aarch64-multiplatform.python3Packages.frozenlist.drvPath")

[Artturin]

python3Packages.cryptography cross will be fixed by #212795, so not a regression

the other 3 build on #245475 so i'm going to merge that pr

[Artturin]

Yeah idk what's going on, doesn't work

hostHost not always correct, maybe the conditional needs tweaking