Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try to make distutils not emit relative paths #10257

Merged
merged 2 commits into from
Aug 6, 2021

Conversation

uranusjr
Copy link
Member

@uranusjr uranusjr commented Aug 1, 2021

#10151 (comment)
#10151 (comment)

To prevent distutils from returning weird paths, try to normalise the target prefix before passing it into distutils.

Not sure how well this would work, so not pinging people for testing yet…

Turns out the original implementation simply had a bug… This should fix the relative path issue (but not the normalisation part).

@uranusjr uranusjr added this to the 21.2.3 milestone Aug 1, 2021
@uranusjr
Copy link
Member Author

uranusjr commented Aug 5, 2021

@mdmintz @msapiro Continuing the conversation in #10151, did you use the built-in venv module, or virtualenv to create the virtual environment? If it is the latter, would it be possible for you to provide the version of virtualenv you created the environment with?

It would also be awesome if you could provide the following outputs for me (from the problematic virtual environment):

import sys

print(sys.prefix)
print(sys.base_prefix)
print(sys.real_prefix)

@msapiro
Copy link

msapiro commented Aug 6, 2021

@mdmintz @msapiro Continuing the conversation in #10151, did you use the built-in venv module, or virtualenv to create the virtual environment? If it is the latter, would it be possible for you to provide the version of virtualenv you created the environment with?

It would also be awesome if you could provide the following outputs for me (from the problematic virtual environment):

import sys

print(sys.prefix)
print(sys.base_prefix)
print(sys.real_prefix)

I have tried to duplicate this issue on two different servers including the one where I see the issue by creating new directories and virtualenvs that mimic the problem layout, but when I do this, I don't see the issue.

When I mimic the activation and cd that my script does when I see the problem, I get

mailman@grizz:/opt/mailman/git$ . ../mm/venv/bin/activate
(venv) mailman@grizz:/opt/mailman/git$ cd mailman
(venv) mailman@grizz:/opt/mailman/git/mailman$ python
Python 3.6.8 (default, Oct  7 2019, 12:59:55) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.prefix)
/opt/mailman/mm/venv
>>> print(sys.base_prefix)
/opt/mailman/mm/venv
>>> print(sys.real_prefix)
/usr
>>> 

When I then do without anything in between

(venv) mailman@grizz:/opt/mailman/git/mailman$ pip --use-feature=in-tree-build install ./

I get

Processing /opt/mailman/git/mailman
Requirement already satisfied: aiosmtpd>=1.4.1 in /opt/mailman/mm/venv/lib/python3.6/site-packages (from mailman==3.3.5b1) (1.4.1)
Requirement already satisfied: alembic>=1.6.2 in /opt/mailman/mm/venv/lib/python3.6/site-packages (from mailman==3.3.5b1) (1.6.2)
Requirement already satisfied: atpublic in /opt/mailman/mm/venv/lib/python3.6/site-packages/atpublic-1.0-py3.6.egg (from mailman==3.3.5b1) (1.0)
Requirement already satisfied: authheaders>=0.9.2 in /opt/mailman/mm/venv/lib/python3.6/site-packages/authheaders-0.12.0-py3.6.egg (from mailman==3.3.5b1) (0.12.0)
Requirement already satisfied: authres>=1.0.1 in /opt/mailman/mm/venv/lib/python3.6/site-packages/authres-1.2.0-py3.6.egg (from mailman==3.3.5b1) (1.2.0)
Requirement already satisfied: click>=8.0.0 in /opt/mailman/mm/venv/lib/python3.6/site-packages (from mailman==3.3.5b1) (8.0.0)
Requirement already satisfied: dnspython>=1.14.0 in /opt/mailman/mm/venv/lib/python3.6/site-packages/dnspython-1.16.0-py3.6.egg (from mailman==3.3.5b1) (1.16.0)
Requirement already satisfied: falcon>=3.0.0 in /opt/mailman/mm/venv/lib/python3.6/site-packages (from mailman==3.3.5b1) (3.0.0)
Requirement already satisfied: flufl.bounce>=4.0 in /opt/mailman/mm/venv/lib/python3.6/site-packages (from mailman==3.3.5b1) (4.0)
Requirement already satisfied: flufl.i18n>=3.2 in /opt/mailman/mm/venv/lib/python3.6/site-packages (from mailman==3.3.5b1) (3.2)
Requirement already satisfied: flufl.lock>=5.1 in /opt/mailman/mm/venv/lib/python3.6/site-packages (from mailman==3.3.5b1) (5.1)
Requirement already satisfied: importlib_resources>=1.1.0 in /opt/mailman/mm/venv/lib/python3.6/site-packages/importlib_resources-1.4.0-py3.6.egg (from mailman==3.3.5b1) (1.4.0)
Requirement already satisfied: gunicorn in /opt/mailman/mm/venv/lib/python3.6/site-packages (from mailman==3.3.5b1) (19.10.0)
Requirement already satisfied: lazr.config in /opt/mailman/mm/venv/lib/python3.6/site-packages/lazr.config-2.2.2-py3.6.egg (from mailman==3.3.5b1) (2.2.2)
Requirement already satisfied: python-dateutil>=2.0 in /opt/mailman/mm/venv/lib/python3.6/site-packages/python_dateutil-2.8.1-py3.6.egg (from mailman==3.3.5b1) (2.8.1)
Requirement already satisfied: passlib in /opt/mailman/mm/venv/lib/python3.6/site-packages/passlib-1.7.2-py3.6.egg (from mailman==3.3.5b1) (1.7.2)
Requirement already satisfied: requests in /opt/mailman/mm/venv/lib/python3.6/site-packages/requests-2.22.0-py3.6.egg (from mailman==3.3.5b1) (2.22.0)
Requirement already satisfied: sqlalchemy<1.4,>=1.2.3 in /opt/mailman/mm/venv/lib/python3.6/site-packages/SQLAlchemy-1.3.11-py3.6-linux-x86_64.egg (from mailman==3.3.5b1) (1.3.11)
Requirement already satisfied: zope.component in /opt/mailman/mm/venv/lib/python3.6/site-packages/zope.component-4.6-py3.6.egg (from mailman==3.3.5b1) (4.6)
Requirement already satisfied: zope.configuration in /opt/mailman/mm/venv/lib/python3.6/site-packages/zope.configuration-4.3.1-py3.6.egg (from mailman==3.3.5b1) (4.3.1)
Requirement already satisfied: zope.event in /opt/mailman/mm/venv/lib/python3.6/site-packages/zope.event-4.4-py3.6.egg (from mailman==3.3.5b1) (4.4)
Requirement already satisfied: zope.interface>=5.0 in /opt/mailman/mm/venv/lib/python3.6/site-packages/zope.interface-5.0.2-py3.6-linux-x86_64.egg (from mailman==3.3.5b1) (5.0.2)
Requirement already satisfied: typing-extensions in /opt/mailman/mm/venv/lib/python3.6/site-packages (from aiosmtpd>=1.4.1->mailman==3.3.5b1) (3.7.4.3)
Requirement already satisfied: attrs in /opt/mailman/mm/venv/lib/python3.6/site-packages (from aiosmtpd>=1.4.1->mailman==3.3.5b1) (20.3.0)
Requirement already satisfied: Mako in /opt/mailman/mm/venv/lib/python3.6/site-packages/Mako-1.1.0-py3.6.egg (from alembic>=1.6.2->mailman==3.3.5b1) (1.1.0)
Requirement already satisfied: python-editor>=0.3 in /opt/mailman/mm/venv/lib/python3.6/site-packages/python_editor-1.0.4-py3.6.egg (from alembic>=1.6.2->mailman==3.3.5b1) (1.0.4)
Requirement already satisfied: dkimpy>=0.7.1 in /opt/mailman/mm/venv/lib/python3.6/site-packages/dkimpy-0.9.5-py3.6.egg (from authheaders>=0.9.2->mailman==3.3.5b1) (0.9.5)
Requirement already satisfied: publicsuffix in /opt/mailman/mm/venv/lib/python3.6/site-packages/publicsuffix-1.1.0-py3.6.egg (from authheaders>=0.9.2->mailman==3.3.5b1) (1.1.0)
Requirement already satisfied: psutil in /opt/mailman/mm/venv/lib/python3.6/site-packages (from flufl.lock>=5.1->mailman==3.3.5b1) (5.7.0)
Requirement already satisfied: zipp>=0.4 in /opt/mailman/mm/venv/lib/python3.6/site-packages/zipp-3.1.0-py3.6.egg (from importlib_resources>=1.1.0->mailman==3.3.5b1) (3.1.0)
Requirement already satisfied: importlib_metadata in /opt/mailman/mm/venv/lib/python3.6/site-packages/importlib_metadata-1.6.0-py3.6.egg (from importlib_resources>=1.1.0->mailman==3.3.5b1) (1.6.0)
Requirement already satisfied: six>=1.5 in /opt/mailman/mm/venv/lib/python3.6/site-packages/six-1.13.0-py3.6.egg (from python-dateutil>=2.0->mailman==3.3.5b1) (1.13.0)
Requirement already satisfied: setuptools in /opt/mailman/mm/venv/lib/python3.6/site-packages (from zope.interface>=5.0->mailman==3.3.5b1) (42.0.0)
Requirement already satisfied: lazr.delegates in /opt/mailman/mm/venv/lib/python3.6/site-packages/lazr.delegates-2.0.4-py3.6.egg (from lazr.config->mailman==3.3.5b1) (2.0.4)
Requirement already satisfied: nose in /opt/mailman/mm/venv/lib/python3.6/site-packages/nose-1.3.7-py3.6.egg (from lazr.delegates->lazr.config->mailman==3.3.5b1) (1.3.7)
Requirement already satisfied: MarkupSafe>=0.9.2 in /opt/mailman/mm/venv/lib/python3.6/site-packages/MarkupSafe-1.1.1-py3.6-linux-x86_64.egg (from Mako->alembic>=1.6.2->mailman==3.3.5b1) (1.1.1)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/mailman/mm/venv/lib/python3.6/site-packages/chardet-3.0.4-py3.6.egg (from requests->mailman==3.3.5b1) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/mailman/mm/venv/lib/python3.6/site-packages/idna-2.8-py3.6.egg (from requests->mailman==3.3.5b1) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/mailman/mm/venv/lib/python3.6/site-packages (from requests->mailman==3.3.5b1) (1.25.7)
Requirement already satisfied: certifi>=2017.4.17 in /opt/mailman/mm/venv/lib/python3.6/site-packages/certifi-2019.9.11-py3.6.egg (from requests->mailman==3.3.5b1) (2019.9.11)
Requirement already satisfied: zope.deferredimport>=4.2.1 in /opt/mailman/mm/venv/lib/python3.6/site-packages/zope.deferredimport-4.3.1-py3.6.egg (from zope.component->mailman==3.3.5b1) (4.3.1)
Requirement already satisfied: zope.deprecation>=4.3.0 in /opt/mailman/mm/venv/lib/python3.6/site-packages/zope.deprecation-4.4.0-py3.6.egg (from zope.component->mailman==3.3.5b1) (4.4.0)
Requirement already satisfied: zope.hookable>=4.2.0 in /opt/mailman/mm/venv/lib/python3.6/site-packages/zope.hookable-5.0.0-py3.6-linux-x86_64.egg (from zope.component->mailman==3.3.5b1) (5.0.0)
Requirement already satisfied: zope.proxy in /opt/mailman/mm/venv/lib/python3.6/site-packages/zope.proxy-4.3.3-py3.6-linux-x86_64.egg (from zope.deferredimport>=4.2.1->zope.component->mailman==3.3.5b1) (4.3.3)
Requirement already satisfied: zope.i18nmessageid in /opt/mailman/mm/venv/lib/python3.6/site-packages/zope.i18nmessageid-5.0.0-py3.6-linux-x86_64.egg (from zope.configuration->mailman==3.3.5b1) (5.0.0)
Requirement already satisfied: zope.schema>=4.9 in /opt/mailman/mm/venv/lib/python3.6/site-packages/zope.schema-4.9.3-py3.6.egg (from zope.configuration->mailman==3.3.5b1) (4.9.3)
Building wheels for collected packages: mailman
  Building wheel for mailman (setup.py) ... done
  Created wheel for mailman: filename=mailman-3.3.5b1-py3-none-any.whl size=2056701 sha256=be2ae61ab007699b9dd62cbf8e6ccb68781a101cbab0ecd999cf9c7f0b81a801
  Stored in directory: /tmp/pip-ephem-wheel-cache-nuef69y6/wheels/ad/ed/68/0253165691441f43732e06b1f3416bad4e66e54120107ebf55
Successfully built mailman
Installing collected packages: mailman
  Attempting uninstall: mailman
    Found existing installation: mailman 3.3.5b1
    WARNING: Value for bin_prefix does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
    distutils: /opt/mailman/git/../mm/venv/bin
    sysconfig: /opt/mailman/mm/venv/bin
    WARNING: Additional context:
    user = False
    home = None
    root = None
    prefix = None
    Uninstalling mailman-3.3.5b1:
      Successfully uninstalled mailman-3.3.5b1
Successfully installed mailman-3.3.5b1

virtualenv --version gives me
15.1.0
but that may not be the virtualenv that built this /opt/mailman/mm/venv/

One other thing which is puzzling. The script that I run actually does the pip --use-feature=in-tree-build install ./ command in six directories - mailman, mailman-hyperkitty, mailmanclient, django-mailman3, hyperkitty and postorius - but the warning only appears on the mailman install.

@mdmintz
Copy link
Contributor

mdmintz commented Aug 6, 2021

@uranusjr The virtual environment was created with virtualenv.

Here's thepip debug output:

pip version: pip 21.2.2 from /Users/michael/.virtualenvs/psb3/lib/python3.7/site-packages/pip (python 3.7)
sys.version: 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24) 
[Clang 6.0 (clang-600.0.57)]
sys.executable: /Users/michael/.virtualenvs/psb3/bin/python
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: utf-8
locale.getpreferredencoding: UTF-8
sys.platform: darwin
sys.implementation:
  name: cpython
'cert' config value: Not specified
REQUESTS_CA_BUNDLE: None
CURL_CA_BUNDLE: None
pip._vendor.certifi.where(): /Users/michael/.virtualenvs/psb3/lib/python3.7/site-packages/pip/_vendor/certifi/cacert.pem
pip._vendor.DEBUNDLED: False
vendored library versions:
  appdirs==1.4.4
  CacheControl==0.12.6
  colorama==0.4.4
  distlib==0.3.2
  distro==1.5.0 (Unable to locate actual module version, using vendor.txt specified version)
  html5lib==1.1
  msgpack==1.0.2 (Unable to locate actual module version, using vendor.txt specified version)
  packaging==21.0
  pep517==0.11.0
  progress==1.5
  pyparsing==2.4.7
  requests==2.26.0
  certifi==2021.05.30
  chardet==4.0.0
  idna==3.2
  urllib3==1.26.6
  resolvelib==0.7.1
  setuptools==44.0.0 (Unable to locate actual module version, using vendor.txt specified version)
  six==1.16.0
  tenacity==8.0.1 (Unable to locate actual module version, using vendor.txt specified version)
  tomli==1.0.3
  webencodings==0.5.1 (Unable to locate actual module version, using vendor.txt specified version)
Compatible tags: 1132
  cp37-cp37m-macosx_10_14_x86_64
  cp37-cp37m-macosx_10_14_intel
  cp37-cp37m-macosx_10_14_fat64
  cp37-cp37m-macosx_10_14_fat32
  cp37-cp37m-macosx_10_14_universal2
  cp37-cp37m-macosx_10_14_universal
  cp37-cp37m-macosx_10_13_x86_64
  cp37-cp37m-macosx_10_13_intel
  cp37-cp37m-macosx_10_13_fat64
  cp37-cp37m-macosx_10_13_fat32

And here's the latest info you wanted:

  python
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/Users/michael/.virtualenvs/psb3/bin/..'
>>> sys.base_prefix
'/Users/michael/.virtualenvs/psb3/bin/..'
>>> sys.real_prefix
'/Library/Frameworks/Python.framework/Versions/3.7'

@uranusjr
Copy link
Member Author

uranusjr commented Aug 6, 2021

Thanks for the additional information! I tried multiple virtualenv versions and was able to reproduce this with virtualenv 13.1.2 (likely earlier). I will still try to fix this warning, but recreating the virtual environment with a newer version of virtualenv should resolve this warning immediately.

Another side note, the output when creating the environment is quite indicative:

$ virtualenv --version
14.0.6
$ virtualenv ../y --python=python3.6
Running virtualenv with interpreter /home/uranusjr/.local/share/pythonup/cmd/python3.6
Using base prefix '/home/uranusjr/.local/share/pythonup/versions/3.6'
New python executable in /home/uranusjr/y/bin/python3.6
Also creating executable in /home/uranusjr/y/bin/python
Installing setuptools, pip, wheel...done.
$ virtualenv --version
13.1.2
$ virtualenv ../y --python=python3.6
Running virtualenv with interpreter /home/uranusjr/.local/share/pythonup/cmd/python3.6
Using base prefix '/home/uranusjr/.local/share/pythonup/versions/3.6'
New python executable in ../y/bin/python3.6
Also creating executable in ../y/bin/python
Installing setuptools, pip, wheel...done.

You can see how 13.1.2 creates a new python executable at a relative path, but 14.0.6 makes the path absolute.

Our own override for the distutils implementation has a bug :)
virtualenv<14 does not normalize sys.prefix correctly, so we need to do
it on our own.
@uranusjr uranusjr marked this pull request as ready for review August 6, 2021 09:00
Copy link
Member

@pradyunsg pradyunsg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand all of it, but looks good to me for the bits that I can understand.

@uranusjr
Copy link
Member Author

uranusjr commented Aug 6, 2021

It’s mostly backward compatibility stuff (some variable/function used before is broken on old version of something so we need to do the same thing with another variable/function). So if you can’t understand the difference, it’s probably right 😆

@uranusjr uranusjr merged commit 0b95378 into pypa:main Aug 6, 2021
@uranusjr uranusjr deleted the locations-relative-prefix branch August 6, 2021 09:56
@uranusjr uranusjr mentioned this pull request Aug 6, 2021
@msapiro
Copy link

msapiro commented Aug 6, 2021

I confirm that this fixes my issue.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants