How to avoid python packages conflicts when changing stdenv?

[nasyxx]

I changed the Python stdenv via:

py39 = pkgs.python39.override {
  self = py39;
  stdenv = pkgs.llvmPackages_latest.stdenv
}

Then, I tried:

> nix-shell -p "py39.withPackages (p: [ p.freezegun ])" --run python
...
running install tests
no Makefile or custom installCheckPhase, doing nothing
pythonCatchConflictsPhase
Found duplicated packages in closure for dependency 'six':
  six 1.15.0 (/nix/store/qdjsq2wmgdxm99d071zviypb7ipplprk-python3.9-six-1.15.0/lib/python3.9/site-packages)
  six 1.15.0 (/nix/store/ldj5ixz2327yi3ak5d3aglc5qzbpd27h-python3.9-six-1.15.0/lib/python3.9/site-packages)

Package duplicates found in closure, see above. Usually this happens if two packages depend on different version of the same dependency.
error: builder for '/nix/store/y2xkql653h1gq4fmbsa4vagbx2m2rjyd-python3.9-freezegun-1.1.0.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/3br135qqmfn9zq98hhg8clpdn14r69sa-python3-3.9.2-env.drv' failed to build

To avoid this error, now I use the overlay below, which ignores the check phase.

...
freezegun = freezegun.overridePythonAttrs (o: {
    doInstallCheck = false;
    doCheck = false;
    checkInputs = [ ];
})
...

Is there any other method to solve this error so that I don't need to ignore the check phase of every error packages?

[FRidh]

The interpreter should be passed to itself, just as when overriding it for any other purpose. That way, the package set will actually use this modified Python.

mypython = let
  python = pkgs.python3.override {
    stdenv = pkgs.llvmPackages_latest.stdenv
    self = python;
  };
in python;

Note however that stdenv is often passed in as an additional argument, because some checks may be done in the package expression. It is also used for non-Python packages such as hooks, which I think is the cause of the collision. Overriding the stdenv like the above will not make stdenv be correct there. In that case, pkgs should be updated as well. In order to avoid rebuilding pkgs with stdenv, you could do a

mypython = let
  stdenv = pkgs.llvmPackages_latest.stdenv;
  python = pkgs.python3.override {
    inherit stdenv;
    self = python;
    pkgs = pkgs // { inherit stdenv; };
  };
in python;

That should get you further. Note, however, this will still fail in case of cross-compilation.

(Haven't tested any of these expressions)

[nasyxx]

Thanks!

But cpython/default.nix doesn't have the pkgs argument.

It has

  , packageOverrides ? (self: super: {})
  , pkgsBuildBuild
  , pkgsBuildHost
  , pkgsBuildTarget
  , pkgsHostHost
  , pkgsTargetTarget

Should I change one of them?

Also, change

mypython = pkgs.python3.override {
  stdenv = pkgs.llvmPackages_latest.stdenv;
  self = mypython;
}

to the let expression,

mypython = let
  python = pkgs.python3.override {
    stdenv = pkgs.llvmPackages_latest.stdenv;
    self = python;
  };
in python;

seems the same.

[FRidh]

Also, change

Correct, I missed you already set self.

Should I change one of them?

So, this is messy. Turns out its actually the pkgs passed into /home/freddy/Code/libraries/nixpkgs/pkgs/development/interpreters/python/default.nix.

mypython = let
  stdenv = pkgs.llvmPackages_latest.stdenv;
  interpreters = pkgs.pythonInterpreters.override {
    pkgs = pkgs // {inherit stdenv};
  };
in interpreters.python38 # if it is 3.8 you want

[nasyxx]

Thanks, but unfortunately, it not works.

I have tested it with:

when using this one:

mypython = let
  stdenv = pkgs.llvmPackages_latest.stdenv;
  interpreters = pkgs.pythonInterpreters.override {
    pkgs = pkgs // {inherit stdenv;};
  };
in interpreters.python38

It still builds with clang7 but the latest clang11.

If I override the python38 again:

mypython = let
  stdenv = pkgs.llvmPackages_latest.stdenv;
  interpreters = pkgs.pythonInterpreters.override {
    pkgs = pkgs // {inherit stdenv;};
  };
  py38 = interpreters.python38.override {
    self = py38;
    inherit stdenv;
  }
in py38

It conflicts again.

[yaitskov]

Hi,

let me refresh the issue.

I also struggle with interpreter replacement and face duplicated toml package.
The bug is reproduced with following schema:

{ pkgs ? import <nixpkgs> {} }:

let
  myPatchedPython = (pkgs.python39.overrideAttrs (old:  {
    src = pkgs.fetchFromGitHub {
      owner = "python";
      repo = "cpython";
      rev = "0a7dcbdb13f1f2ab6e76e1cff47e80fb263f5da0"; # 3.9.5                                                               
      sha256 = "0fw1bdphhahr4syahd6ii732ryfpxzlbpxjc3r81bi05yvhybd3q";
    };

    preBuild=''                                                                                                               
      export LIBFFI_INCLUDEDIR=${pkgs.libffi.dev}/include                                                                     
      export SDKROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX10.12.sdk"                                               
    '';

    commonPreHook = ''                                                                                                        
      export LIBFFI_INCLUDEDIR=${pkgs.libffi.dev}/include                                                                     
      export SDKROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX10.12.sdk";                                              
    '';

    LIBFFI_INCLUDEDIR="${pkgs.libffi.dev}/include";
    CFLAGS="-iwithprefix ${pkgs.libffi.dev}/include";

    buildInputs = old.buildInputs ++ [
      pkgs.libffi
      pkgs.tcl-8_6
      pkgs.tk-8_6
      pkgs.darwin.apple_sdk.frameworks.Tcl
    ];
    doCheck = false;
    LDFLAGS = " -L${pkgs.tcl-8_6}/lib -L${pkgs.tk-8_6}/lib " + old.LDFLAGS;
  }));

  # replace python in packages                                                                                                
  pa-py-pkgs = myPatchedPython.override { self = myPatchedPython; };

in pa-py-pkgs.pkgs.pygogo

log ends with errors:

pythonCatchConflictsPhase
Found duplicated packages in closure for dependency ‘toml’:
toml 0.10.2 (/nix/store/rxkbpax8krb083hmpkl570qgk4m948hv-python3.9-toml-0.10.2/lib/python3.9/site-packages)
toml 0.10.2 (/nix/store/gmv2cfzs5i4ssnk016rhy44dn9j8mh2k-python3.9-toml-0.10.2/lib/python3.9/site-packages)

python version doesn't matter and other python package (e.g. pygit2) are affected but all.

[stale]

I marked this as stale due to inactivity. → More info

[FRidh]

Overriding of parameters is fixed in #169475. Overriding of derivation parameters is harder and does not yet work I think.

[collares]

For what it's worth, the behavior described in #117065 (comment) still happens on Nixpkgs master for me. My use case is master...collares:sage-ccache

[FRidh]

Overriding the stdenv on a per package base should be possible with #173411.

[collares]

@FRidh Thanks a lot for this! I will test it.

With regards to the approach in #117065 (comment), I noticed that some parts of the Python infrastructure use python.pythonForBuild (for example, the interpreter used by the "catch conflicts" hook at https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/python/hooks/default.nix#L96). I think the duplicated packages in the present issue stem from this: buildPlatform and hostPlatform are the same, so the conflict checker runs, but python.stdenv is NOT python.pythonForBuild.stdenv when proceeding as in #117065 (comment), and somehow both Pythons end up in PATH.

I was unable to bypass this. I guess overriding the pythonForBuild stdenv properly would mean rebuilding my whole system, not just the relevant Python parts, but perhaps there is a clean workaround for cases such as overriding stdenv. Do you have any pointers on how to navigate this? Or do you think the best option is just to disable the conflicts check?

[collares]

Related to my previous comment: As a quick-and-dirty test, I removed .pythonForBuild from the two lines at https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/python/hooks/default.nix#L12-L13; after that, the approach in #117065 (comment) works with no conflicts. This is clearly not a solution because of cross-compilation, but hopefully it is a useful observation anyway.

[FRidh]

The idea was to fix this with #117065 (comment), but because of the splicing it is not possible to override derivations. Seems that in the end it broke cross-compilation as well.