Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dependencies specified in extra_requires don't work from 1.5.0 onwards #736

Closed
mridsole opened this issue Jun 17, 2019 · 1 comment · Fixed by #739
Closed

Dependencies specified in extra_requires don't work from 1.5.0 onwards #736

mridsole opened this issue Jun 17, 2019 · 1 comment · Fixed by #739
Assignees
Labels

Comments

@mridsole
Copy link

Example:

setup.py:

from setuptools import setup, find_packages

setup(
    name = 'bad-pex-example',
    version = '0.1',
    python_requires = '>=3.5',
    install_requires = [],
    extras_require = { 'master': [ 'schema>=0.6' ] }
)

bad/__init__.py:

import schema

def entry():
    print('it works')

In version 1.4.9, pex -o bad.pex -e 'bad:entry' '.[master]' produces a working executable. In all later versions (up to current latest 1.6.7) the resulting executable fails to import the dependency:

Traceback (most recent call last):
  File ".bootstrap/pex/pex.py", line 352, in execute
  File ".bootstrap/pex/pex.py", line 284, in _wrap_coverage
  File ".bootstrap/pex/pex.py", line 315, in _wrap_profiling
  File ".bootstrap/pex/pex.py", line 397, in _execute
  File ".bootstrap/pex/pex.py", line 495, in execute_entry
  File ".bootstrap/pex/pex.py", line 509, in execute_pkg_resources
  File ".bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 2384, in resolve
  File "/home/david/work/bad-pex-example/bad/__init__.py", line 1, in <module>
    import schema
ModuleNotFoundError: No module named 'schema'

pex does still get the dependency and does something with it:

❯ pex -o bad.pex -e 'bad:entry' '.[master]' -vvvv
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7f83f8be4908>       
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7f83f8be4240>
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7f83f8b982b0>                                                                                         
pex: crawling link i=0 link=Link('file:///home/david/.pex/build') follow_links=False
pex: crawling link i=0 link=Link('https://pypi.org/simple/schema/') follow_links=False
pex: Building pex :: Resolving distributions :: Fetching file:///home/david/.pex/build/schema-0.7.0-py2.py3-none-any.whl :: Fetching file:///home/david/.pex/build/schema-0.7.0-py2.ppex: Building pex :: Resolving distributions :: Translating /tmp/tmpa2mbu1dq/schema-0.7.0-py2.py3-none-any.whl into distribution                                                     pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7f83f8374208>                                    
pex: crawling link i=0 link=Link('https://pypi.org/simple/contextlib2/') follow_links=False
pex: Building pex :: Resolving distributions :: Fetching file:///home/david/.pex/build/contextlib2-0.5.5-py2.py3-none-any.whl :: Fetching file:///home/david/.pex/build/contextlib2-0pex: Building pex :: Resolving distributions :: Translating /tmp/tmpf5_mo387/contextlib2-0.5.5-py2.py3-none-any.whl into distribution                                                  bad-pex-example==0.1 -> bad-pex-example 0.1
  schema>=0.6; extra == "master" -> schema 0.7.0
  contextlib2==0.5.5 -> contextlib2 0.5.5
pex: Building pex: 470.8ms                                                                                                           
pex:   Resolving interpreters: 0.0ms
pex:   Installing /tmp/tmp1zk6gp55: 209.2ms
pex:     Isolating pex in Chroot(/tmp/tmp6s9psqsi {fs:}): 15.1ms
pex:   Resolving distributions: 260.6ms
pex:     Fetching file:///tmp/tmp1zk6gp55/bad-pex-example-0.1.tar.gz: 0.5ms
pex:       Fetching file:///tmp/tmp1zk6gp55/bad-pex-example-0.1.tar.gz: 0.1ms
pex:     Translating /tmp/tmppx_oldwr/bad-pex-example-0.1.tar.gz into distribution: 169.7ms
pex:       Packaging bad-pex-example: 168.0ms
pex:         Installing /tmp/tmpuosc6ijy: 166.6ms
pex:     Fetching file:///home/david/.pex/build/schema-0.7.0-py2.py3-none-any.whl: 0.4ms
pex:       Fetching file:///home/david/.pex/build/schema-0.7.0-py2.py3-none-any.whl: 0.2ms
pex:     Translating /tmp/tmpa2mbu1dq/schema-0.7.0-py2.py3-none-any.whl into distribution: 0.8ms
pex:     Fetching file:///home/david/.pex/build/contextlib2-0.5.5-py2.py3-none-any.whl: 1.2ms
pex:       Fetching file:///home/david/.pex/build/contextlib2-0.5.5-py2.py3-none-any.whl: 0.4ms
pex:     Translating /tmp/tmpf5_mo387/contextlib2-0.5.5-py2.py3-none-any.whl into distribution: 1.9ms
pex: Vendoring pkg_resources from VendorSpec(key='setuptools', version='40.6.2') @ /home/david/.local/lib/python3.6/site-packages/pex/vendor/_vendored/setuptools
Saving PEX file to bad.pex

and schema still does end up in ~/.pex/install after running the executable:

❯ ls -lah ~/.pex/install
total 20K
drwxrwxr-x 5 david david 4.0K Jun 17 11:42 .
drwxrwxr-x 3 david david 4.0K Jun 17 11:42 ..
drwxrwxr-x 4 david david 4.0K Jun 17 11:42 bad_pex_example-0.1-py3-none-any.whl.09110f078728861fa50039d90343b57aebf726ca
drwxrwxr-x 4 david david 4.0K Jun 17 11:42 contextlib2-0.5.5-py2.py3-none-any.whl.edad6efe1aebf29bea2d8a9e8dab93fbbc2555bc
drwxrwxr-x 4 david david 4.0K Jun 17 11:42 schema-0.7.0-py2.py3-none-any.whl.5b7491a4df1576f5f2912f4158d096cd06d90199

Many thanks!

@jsirois
Copy link
Member

jsirois commented Jun 22, 2019

I repro after:

$ diff -u /tmp/setup.py setup.py 
--- /tmp/setup.py	2019-06-22 15:38:38.599110687 -0600
+++ setup.py	2019-06-22 15:08:11.585058403 -0600
@@ -4,6 +4,7 @@
     name = 'bad-pex-example',
     version = '0.1',
     python_requires = '>=3.5',
+    packages = ['bad'],
     install_requires = [],
     extras_require = { 'master': [ 'schema>=0.6' ] }
 )

The bug was introduced in #582 here:
https://github.com/pantsbuild/pex/blob/fb35498778bf722a33133ed351cea5b47a523c39/pex/resolver.py#L327

The requirement string returned by a distribution for itself will never contain extras that were asked for when installing it. We have that information though in the resolvable the distribution was created from, so this is a one line fix with more lines to add a test.

jsirois added a commit to jsirois/pex that referenced this issue Jun 24, 2019
The change in pex-tool#582 introduced a bug where requested extras were not
preserved for local `setup.py` resolvables. Add failing tests for this
and fix.

Fixes pex-tool#736
jsirois added a commit that referenced this issue Jun 25, 2019
The change in #582 introduced a bug where requested extras were not
preserved for local `setup.py` resolvables. Add failing tests for this
and fix.

Fixes #736
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants