python3: fix overriding of interpreters, closes #163639 #169475
Conversation
|
@Ericson2314 work-around needed for |
|
Note that while this "fixes" the usage of |
Overriding the interpreters did not work correctly. When overriding packages would end up twice in the build time closure: one corresponding to the overridden interpreter and one corresponding to the original interpreter. The reason is that the override was not applied to the interpreters in the spliced package sets.
ofborg
bot
added
10.rebuild-darwin: 0
|
This breaks cross compilation: |
|
I pushed a commit fixing the cross issue in adisbladis@b4fbe2e |
That makes sense. I suppose we could filter derivations out of |
Yeah, that's a far more palatable solution. Tried that out in adisbladis@9203631 and it also works just fine to build a cross compiled environment. |
In the case of cross compilation we don't want pass through build inputs from the cross platform, but we do want to pass on config options.
|
This breaks I used the following Nix code to bisect this: The assertion will fail after this PR (run with |
|
That's unfortunate. Did you test with multiple packages? Seems like we need to revert this then. |
|
@FRidh my original problem was that when overriding python, the old packages were used from the original version. I'm wondering maybe there is a bug in another place and the old behavior was just hiding it? Alternatively, maybe overrides shouldn't be applied for the Build version? Although I'm less convinced of this one. |
|
No, the issue originated exactly because the build Python was not overridden. That is solved now. Unfortunately it is not possible to pass in derivations to the spliced sets because then they're for the wrong purpose (build/host/target). I think it is really difficult to override a build derivation in a spliced package set without globally overriding that derivation (in this case the Python interpreter) with an overlay. |
Yes, the same issue occurs if you replace |
Mindavi
added
the
6.topic: cross-compilation
|
I think the fix to this is as simple as: diff --git a/pkgs/development/interpreters/python/cpython/default.nix b/pkgs/development/interpreters/python/cpython/default.nix
index 4463dc8e9ba..47f9d9a3775 100644
--- a/pkgs/development/interpreters/python/cpython/default.nix
+++ b/pkgs/development/interpreters/python/cpython/default.nix
@@ -85,7 +85,7 @@ let
passthru = let
# When we override the interpreter we also need to override the spliced versions of the interpreter
- inputs' = lib.filterAttrs (_: v: ! lib.isDerivation v) inputs;
+ inputs' = lib.filterAttrs (n: v: ! lib.isDerivation v && n != "passthruFun") inputs;
override = attr: let python = attr.override (inputs' // { self = python; }); in python;
in passthruFun rec {
inherit self sourceVersion packageOverrides; |
|
That fix seems to work. |
see NixOS#169475 (comment) patch by adisbladis Co-authored-by: adisbladis <[email protected]>
see #169475 (comment) patch by adisbladis Co-authored-by: adisbladis <[email protected]> (cherry picked from commit 843b988)
|
Sorry for asking here, but how is this supposed to work? Because I still end up with two versions of the same package, when I do something like: pythonOverlay = final: prev: {
weakPython = final.python311.override {
openssl = true;
openssl_legacy = final.weakOpenssl;
packageOverrides = pyfinal: pyprev: {
pyopenssl = pyprev.pyopenssl.override { openssl = final.weakOpenssl; };
cryptography = pyprev.cryptography.override { openssl = final.weakOpenssl; };
};
self = final.weakPython;
}e.g. building |
|
It seems like it's broken again. |
|
Yes, I'm also seeing it once again: |
|
This change made evaluating cross packages very memory expensive, likely because there is no memoization on the override. I'm inclined to go for a revert, but I have no good idea about how to make the interpreter overridable any other way. In the meantime, if you want to run out of memory, try |
We should probably break out the Python packages factory function and stop conflating overriding the interpreter with overriding the package set. |
|
@adisbladis I'm not really familiar with python development but the memory usage is killing me. Maybe I could tackle this if I had some guidance. |
Overriding the interpreters did not work correctly. When overriding
packages would end up twice in the build time closure: one corresponding
to the overridden interpreter and one corresponding to the original
interpreter. The reason is that the override was not applied to the
interpreters in the spliced package sets.
Description of changes
Things done
sandbox = trueset innix.conf? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)nixos/doc/manual/md-to-db.shto update generated release notes