python2.7-python-otr-1.2.0: Test failure: ImportError: No module named zope.interface

[ckauhaus]

Describe the bug
Cannot build python-otr because of a failing test. I suspect that the cause is an undeclared dependency (but didn't check it yet).

To Reproduce
Steps to reproduce the behavior:

1. nix-build -I nixpkgs=. -A pythonPackages.python-otr on 7241339

Expected behavior
Build output in result

Additional context
See attached log output: python-otr.log

Metadata

• system: "x86_64-linux"
• host os: Linux 4.15.0-65-generic, Ubuntu, 18.04.3 LTS (Bionic Beaver)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.2.2
• channels(ck): "nixos-15.09-15.09.1262.3727911, nixos-16.09-16.09.1943.25f4906da6, nixos-17.09-17.09.3269.14f9ee66e63, nixos-18.09-18.09.2574.a7e559a5504"
• channels(root): "nixpkgs-19.09.1279.5f506b95f9f"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixpkgs

[ckauhaus]

cc @edwtjo

[jonringer]

this is more likely caused by the fact that zope is a namespace package, and in python2, this means that each package would have a "${python.sitePackages}/zope/init.py" file which extends the namespace so that all the packages can be traversed.

The problem is that python2 will fail if you add multiple zope packages (even transitively) because there will be a file collision.

I would personally just disable tests for python, unless zope is imported at runtime, then you would need to do a 'ignoreCollisions' in the nix-expression

[exarkun]

There only appears to be a single zope package installed as a dependency here, so collisions between multiple zope packages doesn't seem to be the issue here. Also, it seems like that would manifest as a collision being reported rather than a test suite failure.

And the zope_interface dependency is correctly declared in propagatedBuildInputs (I think it is correct to put python dependencies in either buildInputs or propagatedBuildInputs).

The failure looks a lot like another one I am trying to track down which appears to have been introduced in 90be4c2.

And ... indeed. I can reproduce the reporter's failure on that revision but the build succeeds on the prior revision. I don't understand why though.

[exarkun]

The zope.interface site-packages directory isn't being added as a user site directory. This prevents its pth file from being processed and this makes it unimportable.

I'm not sure why it isn't added as a user site directory, though. PYTHONNOUSERSITE=1 is set but it was also set before so I don't know if it matters.

[exarkun]

Here's a minimal example that reproduces the problem - https://github.com/exarkun/minimal-usersite-repro-nixos. I still haven't managed to untangle how Python packaging works in Nixpkgs well enough to see why the referenced revision matters, though.

I hope someone with more experience (@FRidh ?) can take a look. Right now it seems like all Python packages that use namespace packages are broken?

[exarkun]

This can be demonstrated even more easily, actually.

nix-shell -p python2.pkgs.zope_interface --run 'python -c "import zope.interface"'

This works until 90be4c2 and then is broken.

[FRidh]

No, it's only this package and I have no idea why.

[exarkun]

When you say "this package" I guess you mean zope.interface?

[jonringer]

zope_interface is the package, zope.interface is a module

[exarkun]

Maybe it was never reasonable to expect nix-shell -p python2.pkgs.zope_interface --run 'python -c "import zope.interface"' to work at all? A more reasonable package expression is perhaps something like python2.withPackages (ps: [ ps.zope_interface ]) and, interestingly, this still works - though I still don't quite know why.

Is that really a more reasonable expression to use? If so, does it have any bearing on this problem at all? Is there some way to get that kind of python environment created for use when building packages that run into this issue?

It seems like setuptools-check-hook picks the base Python package and runs the Python executable from that without setting up a Python environment. Somehow this kinda mostly sorta works but it's different from the recommended way to actually use Python on NixOS.

Maybe it should change? Or, maybe I've seen some noise about setuptools-check-hook being bad and old and on its way to deprecation and there is some alternative that packages can already switch to now which works in this case.

[FRidh]

Yes, withPackages should be used and not nix-shell -p.

[exarkun]

If there are any further hints you can give, I'd really appreciate them. I've been banging my head on this for over three full days now and I really need to get it fixed.

[exarkun]

Okay, so it still seems to me like setuptools-check-hook uses the wrong Python environment to run the test suite and it only works by accident for most packages.

I fixed another package with this diff

diff --git a/nix/eliot.nix b/nix/eliot.nix
index f6d6b3061..c76864101 100644
--- a/nix/eliot.nix
+++ b/nix/eliot.nix
@@ -1,5 +1,5 @@
 { lib, buildPythonPackage, fetchPypi, zope_interface, pyrsistent, boltons
-, hypothesis, testtools, pytest }:
+, hypothesis, testtools, pytest, python }:
 buildPythonPackage rec {
   pname = "eliot";
   version = "1.7.0";
@@ -19,6 +19,14 @@ buildPythonPackage rec {
   checkInputs = [ testtools pytest hypothesis ];
   propagatedBuildInputs = [ zope_interface pyrsistent boltons ];
 
+  dontUseSetuptoolsCheck = true;
+  installCheckPhase =
+  let
+    pyenv = python.withPackages (ps: propagatedBuildInputs ++ checkInputs);
+  in ''
+  ${pyenv}/bin/pytest
+  '';
+
   meta = with lib; {
     homepage = https://github.com/itamarst/eliot/;
     description = "Logging library that tells you why it happened";

but I don't yet see how to apply this more generally since setuptools-check-hook.sh doesn't know what checkInputs or propagatedBuildInputs are, and also it is a shell script so it can't exactly use python.withPackages.

I guess the plumbing at some higher level needs to be fixed. It would be nice to have some indication that this is the right direction, though, before I go very much further on it (and the per-package fix is going to remove the immediate blocker for me).

[FRidh]

Okay, so it still seems to me like setuptools-check-hook uses the wrong Python environment to run the test suite and it only works by accident for most packages.

That's quite a claim :) During a build the environment is composed using PYTHONPATH, which does not extend the site packages listing and thus does not consider .pth files. A possible solution is to go for NIX_PYTHONPATH instead if that is relevant (which is a change I intend to make but which also has a downside). Another issue that matters is the current working directory when running tests. By default that is the build directory, and that sometimes causes issues as well.

[exarkun]

Okay, so it still seems to me like setuptools-check-hook uses the wrong Python environment to run the test suite and it only works by accident for most packages.

That's quite a claim :) During a build the environment is composed using PYTHONPATH, which does not extend the site packages listing and thus does not consider .pth files.

Since .pth files are part of the way the Python import machinery works, not having support for them during the build seems like a legitimate shortcoming of Python support in Nixpkgs and so the "accident" of most Python packages working in Nixpkgs seems legitimate - they will be broken if they use a standard, supported Python feature.

That said, let me apologize if I gave any offense by describing the situation that way. I know Python's build, packaging, and import systems are complex and contain many edge cases and were all designed for systems not very like Nixpkgs. I know a lot of work has gone into Nixpkgs' support of Python and I'm just trying to get to the bottom of why this particular case doesn't work so I can continue to leverage the amazing power of Nixpkgs in my projects. So, again, my apologies about my choice of language and my thanks for your efforts in general and on this issue specifically.

A possible solution is to go for NIX_PYTHONPATH instead if that is relevant (which is a change I intend to make but which also has a downside).

Is the downside documented anywhere so I could take a look? Or do you feel like explaining here? Maybe I can contribute to that effort somehow. I've seen NIX_PYTHONPATH in the course of investigating this issue and I have some general idea of what it does but I still don't have the whole Nixpkgs Python build system in my head (probably not anything close to it).

Another issue that matters is the current working directory when running tests. By default that is the build directory, and that sometimes causes issues as well.

Indeed. Since this depends on details of the particular package being tested, I imagine Nixpkgs would need to address it by allowing packages to declare how they need their tests run.

[FRidh]

If you search for NIX_PYTHONPATH on the tracker you will find a few issues describing what it is for. In short, it is a PATH variable like PYTHONPATH, but instead of adding to sys.path, it adds with site.addsitedir so .pth are dealt with. It is implemented in a sitecustomize.py originally as a substitute for PYTHONHOME. It is being unset automatically when Python starts up so it does not leak. Note it also has issues, specifically regarding sys.prefix. Probably better would be to revert to PYTHONHOME for python.withPackages, but having a flag for deciding whether it should be unset (default is yes) or not. NIX_PYHONPATH could still be used during the build process as that one seems most fit for it.

[ckauhaus]

@cillianderoiste any idea?

[aanderse]

Is there anything required other than time and research to get this resolved? This package breaking has suddenly put me in a very bad spot at the moment so if there is anything I can do to provide support, etc... please let me know.

cc @anyonewhocanpossiblyhelprightnow

[jonringer]

You can always use the virtualenv escape hatch: https://nixos.wiki/wiki/Python#Emulating_virtualenv_with_nix-shell

You will need to do pip install <pkgs> to pull down what you are care about, but for the affected packages you should get an environment with them included.

I do this a lot for local development:

# shell.nix
let
  pkgs = import <nixpkgs> {};
  python = pkgs.python37;
  pythonPackages = python.pkgs;
in

mkShell {
  name = "pip-env";
  buildInputs = with pythonPackages; [
    # System requirements.
    readline

    # Python requirements (enough to get a virtualenv going).
    h5py
    jupyter
    requests
    typed-ast
    pandas
    virtualenvwrapper

    libffi
    openssl
    gcc
  ];
  src = null;
  shellHook = ''
    # Allow the use of wheels.
    SOURCE_DATE_EPOCH=$(date +%s)
    # Augment the dynamic linker path
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${pkgs.ncurses}/lib
    VENV=master37env
    if test ! -d $VENV; then
      virtualenv $VENV
    fi
    source ./$VENV/bin/activate

    export PYTHONPATH=`pwd`/$VENV/${python.sitePackages}/:$PYTHONPATH
  '';
}

[jonringer]

generally, the more annoying packages to build such as pandas or tensorflow i get from nix, then I'm fine with vanilla packages through venv

[aanderse]

Thanks @jonringer. Good for now, though still hoping this issue will be resolved in a timely manner...

[jonringer]

python2 is sunsetting in 2 weeks, so it will.... just give it time :)

[exarkun]

Sad to say there are still many users still firmly based on Python 2.7 and this is not likely to change for some while to come (let alone in the next 2 weeks).

Are there official plans to drop Python 2 support from nixpkgs at some point? I hope not.

Absent such plans, this seems like an important issue to fix. Presently a large chunk of Python 2 libraries are unavailable from Nixpkgs either due directly to this issue or because of transitive dependencies that are affected.

[jonringer]

probably be up to @FRidh to make the final call.

Personally, the only way I see it getting dropped is if there's active severe CVE's out for it, which jeopardize users. Otherwise, it will probably go into python3.5/python3.6 territory where we don't recurse into the package set and have hydra build it anymore. Essentially like a "deprecated" state, but still available.

python package set already has a fair amount of branching logic around python2 vs python. I know personally, I've stopped freezing python2 compatible packages, and instead, just disable them for python2 now.

$ cat pkgs/top-level/python-packages.nix | grep " if isP" | wc -l
30
$ cat pkgs/development/python-modules/**/*.nix | grep "isPy" | wc -l
1583

[jonringer]

from a package manager standpoint, maintaining python2 is kind of a PITA due to all the backports and a significant number of upstream packages have dropped support for python2 altogether

[exarkun]

from a package manager standpoint, maintaining python2 is kind of a PITA due to all the backports

Sure. That is reasonable and understandable. Note that from a package user standpoint, losing Python 2 is kind of a PITA due to inability to switch to Python 3 because of unported software. So there is pain on both sides.

If Nixpkgs is going to drop Python 2 ASAP it would be nice for package users to know so we don't waste a huge amount of effort trying to make it work only to have that effort thrown in the trash.

and a significant number of upstream packages have dropped support for python2 altogether

Personally, I wouldn't expect Nixpkgs to try to offer updated Python 2 packages for packages where upstream has dropped Python 2 support. I would certainly hope that Nixpkgs would keep the last Python 2-supporting version of such libraries, though, until Python 2 support is completely removed (which I do assume will happen at some point, I just hope it's not too soon).

I don't know how feasible this is though, since I know the infrastructure generally merges the definitions of Python 2 and Python 3 packages in a single expression. This is increasingly going to be a problem for anyone who still needs to use Python 2, though, and I hope that Nixpkgs can try to ease this pain rather than exacerbate it.

[FRidh]

Unless you are willing to ensure your packages will still function, don't count on them to work anymore. As Jon mentioned, leaf packages now often get disabled. It is up to the community members to ensure they function.

…

On Wed, 18 Dec 2019, 20:51 Jean-Paul Calderone, ***@***.***> wrote: from a package manager standpoint, maintaining python2 is kind of a PITA due to all the backports Sure. That is reasonable and understandable. Note that from a package *user* standpoint, losing Python 2 is kind of a PITA due to inability to switch to Python 3 because of unported software. So there is pain on both sides. If Nixpkgs is going to drop Python 2 ASAP it would be nice for package users to know so we don't waste a huge amount of effort trying to make it work only to have that effort thrown in the trash. and a significant number of upstream packages have dropped support for python2 altogether Personally, I wouldn't expect Nixpkgs to try to offer updated Python 2 packages for packages where upstream has dropped Python 2 support. I would certainly hope that Nixpkgs would keep the last Python 2-supporting version of such libraries, though, until Python 2 support is completely removed (which I *do* assume will happen at some point, I just hope it's not too soon). I don't know how feasible this is though, since I know the infrastructure generally merges the definitions of Python 2 and Python 3 packages in a single expression. This is increasingly going to be a problem for anyone who still needs to use Python 2, though, and I hope that Nixpkgs can try to ease this pain rather than exacerbate it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#74619?email_source=notifications&email_token=AAQHZ33OK25RAVJNVEU4O6TQZJ5M5A5CNFSM4JS6K7SKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHHJBUY#issuecomment-567185619>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQHZ35KOOSGQ7QLXIK77YLQZJ5M5ANCNFSM4JS6K7SA> .

[exarkun]

So it would be fair to say that Python 2 in Nixpkgs is already unsupported?

[FRidh]

That I would not say. It is a community project, and so whether something functions depends on whether there are people putting in effort to keep it working.

…

On Wed, 18 Dec 2019, 21:08 Jean-Paul Calderone, ***@***.***> wrote: So it would be fair to say that Python 2 in Nixpkgs is *already* unsupported? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#74619?email_source=notifications&email_token=AAQHZ367IHNRAPPNMMRQKK3QZJ7NTA5CNFSM4JS6K7SKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHHKZZA#issuecomment-567192804>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQHZ37MMYQENWDL5PR2C4DQZJ7NTANCNFSM4JS6K7SA> .

[jonringer]

there's a compromise with trying to have all the packages up-to-date, and a package supporting all platforms, architectures, and toolchain versions.

unless someone is concerned about a package still being available (like @timokau and sage), then there's likely no one to ensure that a python2 package set will be available for a given package.

[jonringer]

Personally, i have little empathy for projects that still rely on python2. In 2011, they made it official that there will not be a python2.8 with PEP 404. This means that there was ~4 years given to respond, then it was extended on a 5 year LTS loan.

Not vectoring a project to do the eventual switch seems like a grave oversight.

[timokau]

For what it's worth I can't wait until I don't have to support python2 anymore either. The sage switch is underway.

[exarkun]

That I would not say. It is a community project, and so whether something
functions depends on whether there are people putting in effort to keep it
working.

Great. Do you have suggestions about how I can most effectively direct my efforts towards a particular set of Python packages? Should I add myself as a maintainer to packages that I am interested in? Do listed maintainers get a chance to update packages before they're disabled due to being broken by other changes? Or is there some other workflow? I'm happy to read through some contribution documentation on this topic, if you can refer me to it. I haven't come across anything that goes into relevant specifies about how the Python part of Nixpkgs is maintained in my reading so far.

For what it's worth I can't wait until I don't have to support python2 anymore either. The sage switch is underway.

Me too! I have been working to get away from Python 2 for years now. I'm sure everyone is aware of what a huge undertaking this can be for folks who are more than hobbiests with a handful of simple scripts. My current ongoing interest in Python 2 is driven by a 70k+ LOC project that was written before Python 3 ever existed and has been maintained exclusively through volunteer efforts since the company that original wrote it went out of business (and generously open sourced their work instead of condemning it to the garbage heap of failed proprietary projects). There's only so fast a small community of volunteers can make progress on the porting effort. I will be ecstatic when we can leave Python 2 behind ... but we're just not there yet.

[FRidh]

Great. Do you have suggestions about how I can most effectively direct my efforts towards a particular set of Python packages? Should I add myself as a maintainer to packages that I am interested in? Do listed maintainers get a chance to update packages before they're disabled due to being broken by other changes? Or is there some other workflow? I'm happy to read through some contribution documentation on this topic, if you can refer me to it. I haven't come across anything that goes into relevant specifies about how the Python part of Nixpkgs is maintained in my reading so far.

Add yourself as maintainer to the packages you want to maintain. Whenever there is a change affecting it, you will be by pinged by our bot, unless the change is not made through a PR but directly pushed to a branch. So, sometimes it may happen you'll need to fix expressions.

[exarkun]

Add yourself as maintainer to the packages you want to maintain. Whenever there is a change affecting it, you will be by pinged by our bot, unless the change is not made through a PR but directly pushed to a branch. So, sometimes it may happen you'll need to fix expressions.

Thanks. As soon as I figure out how to generate a list of our Python dependencies I'll send a PR. :)

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[ckauhaus]

I think this one is obsolete now.

[exarkun]

Can you explain your reasoning for those not so carefully following Python support in nixpkgs? Thank you.

[exarkun]

@ckauhaus ping

[ckauhaus]

I don't see people really invested in 2.7 anymore. No one came up with a fix for half a year. As Python 2.7 has been deprecated upstream, I doubt this is going to change.

Discussion continues in https://discourse.nixos.org/t/upcoming-python-breakage/7554

[fincham]

I'm running in to the same problem trying to run graphite_api on 20.03. It seems like this just isn't possible until the next release then where it moves to Python 3? :(

[FRidh]

Someone else having this issue not using Nix pypa/pip#8505.