Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#egg=name-version dep breaks package install in build-system.requires #252

Closed
ghost opened this issue Apr 13, 2019 · 26 comments
Closed

#egg=name-version dep breaks package install in build-system.requires #252

ghost opened this issue Apr 13, 2019 · 26 comments

Comments

@ghost
Copy link

ghost commented Apr 13, 2019

I have a problem that might be a packaging issue with my package (wobblui) or a pip bug.

The package itself in direct install works:

$ pip3 install --user -U https://github.com/wobblui/wobblui/archive/master.zip
Collecting https://github.com/wobblui/wobblui/archive/master.zip
  Downloading https://github.com/wobblui/wobblui/archive/master.zip
     / 2.5MB 25.1MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied, skipping upgrade: PySDL2>=0.9.6 in /home/jonas/.local/lib/python3.7/site-packages (from wobblui==2018.8.post2) (0.9.6)
Requirement already satisfied, skipping upgrade: Pillow in /usr/lib64/python3.7/site-packages (from wobblui==2018.8.post2) (5.3.0)
Requirement already satisfied, skipping upgrade: nettools@ https://github.com/JonasT/nettools/archive/master.zip from https://github.com/JonasT/nettools/archive/master.zip in /home/jonas/.local/lib/python3.7/site-packages (from wobblui==2018.8.post2) (0.1)
Building wheels for collected packages: wobblui
  Building wheel for wobblui (PEP 517) ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-qaimgd2r/wheels/a2/25/f0/8a0ee8d22f58b8119c3ed947c2f850ffbfefefec21f4c4d7d1
Successfully built wobblui
Installing collected packages: wobblui
Successfully installed wobblui-2018.8.post2

However, 1. download test2-example.zip, 2. extract it, and 3. install it with pip, and you'll see that for some reason, as a build-system.requires dependency ("pyproject.toml") it doesn't work:

~/test2$ pip3 install --user -U .
Processing /home/jonas/test2
  Installing build dependencies ... error
  Complete output from command /usr/bin/python3 /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-bfw75sjy/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel Cython https://github.com/wobblui/wobblui/archive/master.zip#egg=wobblui-master:
  Collecting setuptools
    Using cached https://files.pythonhosted.org/packages/c8/b0/cc6b7ba28d5fb790cf0d5946df849233e32b8872b6baca10c9e002ff5b41/setuptools-41.0.0-py2.py3-none-any.whl
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/96/ba/a4702cbb6a3a485239fbe9525443446203f00771af9ac000fa3ef2788201/wheel-0.33.1-py2.py3-none-any.whl
  Collecting Cython
    Using cached https://files.pythonhosted.org/packages/ab/1b/85223a9d7d39ea8d0dec76785ef5e2aa7a82a3eaa92cb7becd2a64cef663/Cython-0.29.6-cp37-cp37m-manylinux1_x86_64.whl
  Collecting wobblui-master from https://github.com/wobblui/wobblui/archive/master.zip#egg=wobblui-master
    Downloading https://github.com/wobblui/wobblui/archive/master.zip
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
    Generating metadata for package wobblui-master produced metadata for project name wobblui. Fix your #egg=wobblui-master fragments.
  Collecting nettools@ https://github.com/JonasT/nettools/archive/master.zip from https://github.com/JonasT/nettools/archive/master.zip (from wobblui)
    Downloading https://github.com/JonasT/nettools/archive/master.zip
  Collecting PySDL2>=0.9.6 (from wobblui)
  Collecting Pillow (from wobblui)
    Using cached https://files.pythonhosted.org/packages/c1/e6/ce127fa0ac17775bc7887c432ffe945c49ae141f01b477b7cd5e63b16bb5/Pillow-6.0.0-cp37-cp37m-manylinux1_x86_64.whl
  Building wheels for collected packages: wobblui, wobblui
    Building wheel for wobblui (PEP 517): started
    Building wheel for wobblui (PEP 517): still running...
    Building wheel for wobblui (PEP 517): still running...
    Building wheel for wobblui (PEP 517): still running...
    Building wheel for wobblui (PEP 517): still running...
    Building wheel for wobblui (PEP 517): finished with status 'done'
    Stored in directory: /tmp/pip-ephem-wheel-cache-5v7g1hy2/wheels/a2/25/f0/8a0ee8d22f58b8119c3ed947c2f850ffbfefefec21f4c4d7d1
    Building wheel for wobblui (PEP 517): started
    Error [Errno 2] No such file or directory: '/tmp/pip-install-zjb0_4u0/wobblui-master': '/tmp/pip-install-zjb0_4u0/wobblui-master' while executing command /usr/bin/python3 /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmps99pbdre
    Building wheel for wobblui (PEP 517): finished with status 'error'
    Failed building wheel for wobblui
    Running setup.py clean for wobblui
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-zjb0_4u0/wobblui/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
  
    ----------------------------------------
    Failed cleaning build dir for wobblui
  Successfully built wobblui
  Failed to build wobblui
  Building wheels for collected packages: nettools
    Building wheel for nettools (setup.py): started
    Building wheel for nettools (setup.py): finished with status 'done'
    Stored in directory: /tmp/pip-ephem-wheel-cache-5v7g1hy2/wheels/a1/76/b5/967193e116a1638584f2399b3f4323e2940afea625db569613
  Successfully built nettools
  Could not build wheels for wobblui which use PEP 517 and cannot be installed directly
  
  ----------------------------------------
Command "/usr/bin/python3 /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-bfw75sjy/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel Cython https://github.com/wobblui/wobblui/archive/master.zip#egg=wobblui-master" failed with error code 1 in None

python version: 3.7.3
pip version: 19.0.3

Anyone happens to know why it's broken?

Does someone possibly see whether the package (which is openly available here: https://github.com/wobblui/wobblui ) has a problem causing this, or whether this is a pip bug?

@ghost ghost changed the title Install fails with package in build-system.requires that on itself is installable Install fails with package in build-system.requires that on its own is installable Apr 14, 2019
@benoit-pierre
Copy link
Member

Do you have setuptools installed? I have the same error with a clean virtualenv (with only an up-to-date pip) because your nettools package does not have a pyproject.pytoml declaring its build time requirements, and so needs setuptools to already be available.

Also, you should use PEP 508 direct URL requirements (instead of the old egg notation):

 pyproject.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git i/pyproject.toml w/pyproject.toml
index 3e32eb4..b5cf268 100644
--- i/pyproject.toml
+++ w/pyproject.toml
@@ -1,3 +1,3 @@
 [build-system]
-requires = ["setuptools", "wheel", "Cython", "https://github.com/wobblui/wobblui/archive/master.zip#egg=wobblui-master"]
+requires = ["setuptools", "wheel", "Cython", "wobblui @ https://github.com/benoit-pierre/wobblui/archive/using_fixed_nettools.zip#"]
 build-backend = "setuptools.build_meta"

Note the workaround the pip bug you already encountered (pypa/pip#6405).

Patched wobblui and nettools here:

But this will just kick the can a little further down the road as PySDL2 is similarly dependent on setuptools being already installed, so you should really make sure pip, setuptools and wheel are installed and up-to-date for best results.

@ghost
Copy link
Author

ghost commented Apr 14, 2019

@benoit-pierre setuptools are definitely installed (in system python I mean) is this possibly a build isolation problem? Because this worked fine before I moved wobblui and the main project (test2-example is a stripped down version of that) to pyproject.toml.

Edit: for what it's worth, just to show it:

$ python3
Python 3.7.3 (default, Mar 27 2019, 13:41:07) 
[GCC 8.3.1 20190223 (Red Hat 8.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>> 

As for PEP 508 URLs, once I used them it broke for me. Is the # at the end the workaround to make them work?

@ghost
Copy link
Author

ghost commented Apr 14, 2019

I am just wondering, to support older pips and such I also specify nettools as setup_requires entry in addition. Will that possibly cause easy_install to run unintentionally in the build isolated thing (instead of outside as I assume is needed for non-pyproject.toml legacy things) and break down? Could that be related?

@benoit-pierre
Copy link
Member

benoit-pierre commented Apr 14, 2019

Using the patched example from above, from a clean virtualenv (well technically a venv) with pip/setuptools/wheel installed, yes, it works fine.

As for PEP 508 URLs, once I used them it broke for me. Is the # at the end the workaround to make them work?

Yes.

@ghost
Copy link
Author

ghost commented Apr 14, 2019

Yes but it should normally work without nettools being patched, right? I don't understand how it is intentional that it wouldn't work for that, with setuptools present in system python. (which it is on this system!) What bothers me is that as you already mention, other legacy packages like PySDL2 could potentially cause the same issues so it would be interesting for me to understand why it doesn't work

@benoit-pierre
Copy link
Member

IMHO, it's a loosing battle to try to support older setuptools/pip versions. And setup_requires is broken by design: you can't declare the version of setuptools needed in setup_requires because by the time it's handled setuptools has already been imported...

@benoit-pierre
Copy link
Member

Yes but it should normally work without nettools being patched, right?

Yes, it seems to work with a clean venv with pip / setuptools / wheel.

@benoit-pierre
Copy link
Member

If it does not for you, you'll have to provide the full output of running with -v.

@ghost
Copy link
Author

ghost commented Apr 14, 2019

Did you test it without a pyproject.toml to nettools? I will do so now, because that is the case that is interesting to me

@benoit-pierre
Copy link
Member

Yes, with your unpatched example.

@benoit-pierre
Copy link
Member

It's still installing, but building wobblui, so I'm past your error, no?

@ghost
Copy link
Author

ghost commented Apr 14, 2019

Ooohhh you mean everything works for you fine in a clean venv with setuptools, I misread that sorry 😄 well that is interesting! For what it's worth, I think it happens past the cythonization of wobblui so you may have to wait a couple of minutes (sorry 😢 I overdid Cython use a little because android devices are so underpowered)

@benoit-pierre
Copy link
Member

I'm getting an error with wobblui, but I think it's the use of an egg fragment that could be the issue:

    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-5tmkvlxy/wobblui/setup.py'

But the temporary directory basename is really wobblui-master:

Removing source in /tmp/pip-install-5tmkvlxy/wobblui-master

@ghost
Copy link
Author

ghost commented Apr 14, 2019

Hmm okay, is that something I could/should fix? For what it's worth my -v build is running again so I can show you the full log (this time with tee, just lost the output of the last one accidentally closing the terminal window searching for a select all 😂 lol)

@ghost
Copy link
Author

ghost commented Apr 14, 2019

Ok here is the full output of test2 being installed with everything unchanged:

log.txt

My impression appears to be that something here might be non-deterministic. The last run I failed to save it ended with import setuptools failing, but now I got this instead: Error [Errno 2] No such file or directory: '/tmp/pip-install-i1hb_v78/wobblui-master': '/tmp/pip-install-i1hb_v78/wobblui-master' while executing command /usr/bin/python3 /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpwygn9_pc

@benoit-pierre
Copy link
Member

Note that pip and setuptools interpret the egg fragment differently: setuptools expect PROJECT_NAME-VERSION, but pip only wants PROJECT_NAME.

So again, fix your requirements to use PROJECT_NAME @ URL# instead.

@ghost
Copy link
Author

ghost commented Apr 14, 2019

I dug a bit into the code, and currently install_requires is actually project_name @ URL#egg=project_name-version, so it's already PEP508! (the requirements.txt gets transformed in setup.py)

However, I assume from your very helpful hints that the #egg=project_name-version is what pip doesn't like. I'll change it to #egg=project_name and see what happens

Edit: oops it's not in pyproject.toml, missed that one. let's change that as well

Edit: corrected from #egg-name to #egg

@benoit-pierre
Copy link
Member

Unless I'm mistaken, egg fragments are something that's was added in setuptool, not formalized in any PEP.

Anyway, this work:

  • in test2:
 pyproject.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git i/pyproject.toml w/pyproject.toml
index 3e32eb4..656565e 100644
--- i/pyproject.toml
+++ w/pyproject.toml
@@ -1,3 +1,3 @@
 [build-system]
-requires = ["setuptools", "wheel", "Cython", "https://github.com/wobblui/wobblui/archive/master.zip#egg=wobblui-master"]
+requires = ["setuptools", "wheel", "Cython", "wobblui @ https://github.com/benoit-pierre/wobblui/archive/working.zip#"]
 build-backend = "setuptools.build_meta"
  • in wobblui:
 pyproject.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git i/pyproject.toml w/pyproject.toml
index 3e32eb4..656565e 100644
--- i/pyproject.toml
+++ w/pyproject.toml
@@ -1,3 +1,3 @@
 [build-system]
-requires = ["setuptools", "wheel", "Cython", "https://github.com/wobblui/wobblui/archive/master.zip#egg=wobblui-master"]
+requires = ["setuptools", "wheel", "Cython", "wobblui @ https://github.com/benoit-pierre/wobblui/archive/working.zip#"]
 build-backend = "setuptools.build_meta"

@ghost
Copy link
Author

ghost commented Apr 14, 2019

Yup, seems it was purely the #egg=name-version's fault. It works!! Thank you so much 🥂 🎉 🎉

for what it's worth, maybe other packages might also run into this upgrading to pyproject.toml & pip 19? so that might be worth some investigation - but I'm good as far as I'm concerned, you saved my day!

@benoit-pierre
Copy link
Member

I don't think it's related to PEP 517 / 518 per se, but a general issue with egg fragments handling.

@ghost
Copy link
Author

ghost commented Apr 14, 2019

For what it's worth, I had this in place without pyproject.toml and under pip 18, and it worked without issue. So that's why I assumed that maybe there is something connected there that broke it (some sort of regression?) but I don't actually know. Thanks again for your help!

@ghost ghost changed the title Install fails with package in build-system.requires that on its own is installable #egg=name-version dep breaks package install in build-system.requires Apr 14, 2019
@ghost
Copy link
Author

ghost commented Apr 14, 2019

@benoit-pierre should I leave this open for further investigations or something? or should I close this? (since all that affects me personally is now resolved, I just removed the #egg=name-version things and everything works splendidly!)

@benoit-pierre
Copy link
Member

I think this can be closed as the issue is tracked here: pypa/pip#5271.

@ncoghlan
Copy link
Member

@Jonast Pre-PEP-517, the build requirements were either being picked up from the installation environment (since the build wouldn't have been done in an isolated directory), or else they were getting installed implicitly by setuptools with easy_install instead of by pip (and hence the egg fragment in the URL would have been interpreted differently).

@ghost
Copy link
Author

ghost commented Apr 17, 2019

@ncoghlan yes but unless you put really big warning signs all over the place or just fix it, probably a lot more people will run into this. So something should be done IMHO, unless you want many confused people who had used #egg-name and try to upgrade to PEP 518 and suddenly it doesn't work

@ncoghlan
Copy link
Member

@Jonast Aye, the closure here is only because you encountered the pip bug that @benoit-pierre linked, and we don't want to fragment that discussion.

However, I'll add a note over there explicitly pointing out that more projects may start encountering this as they switch from installing their build dependencies with easy_install (pre-PEP-517) to installing them with pip (post-PEP-517)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants