pip install break with 20.0.6 within virtualenv

[La0]

Issue

Installing black from source in a new virtualenv does not work anymore, and throws an error No module named pip

Environment

Docker image python:3.7

Package            Version
------------------ -------
appdirs            1.4.3  
distlib            0.3.0  
filelock           3.0.12 
importlib-metadata 1.5.0  
pip                20.0.2 
setuptools         45.2.0 
six                1.14.0 
virtualenv         20.0.6 
wheel              0.34.2 
zipp               3.0.0 

Output of the virtual environment creation

root@db1408fc1aec:/# virtualenv x
created virtual environment CPython3.7.6.final.0-64 in 414ms
  creator CPython3Posix(dest=/x, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/tmp/tmp9u90e1vn/seed-app-data/v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

Running the following script in a python:3.7 docker image, but the same would work when pinning virtualenv==20.0.5

[gaborbernat]

This is upstream bug pypa/pip#7778

[La0]

Actually pip install black works OK, it's only when installing from source that it breaks:

root@6fad6b6df528:/# pip install virtualenv
Collecting virtualenv
  Downloading virtualenv-20.0.6-py2.py3-none-any.whl (4.6 MB)
     |████████████████████████████████| 4.6 MB 3.2 MB/s 
Collecting appdirs<2,>=1.4.3
  Downloading appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Collecting filelock<4,>=3.0.0
  Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting six<2,>=1.9.0
  Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting distlib<1,>=0.3.0
  Downloading distlib-0.3.0.zip (571 kB)
     |████████████████████████████████| 571 kB 46.7 MB/s 
Collecting importlib-metadata<2,>=0.12; python_version < "3.8"
  Downloading importlib_metadata-1.5.0-py2.py3-none-any.whl (30 kB)
Collecting zipp>=0.5
  Downloading zipp-3.0.0-py3-none-any.whl (4.8 kB)
Building wheels for collected packages: distlib
  Building wheel for distlib (setup.py) ... done
  Created wheel for distlib: filename=distlib-0.3.0-py3-none-any.whl size=340427 sha256=7f3eea1823f9c13faec9e1c9770dbeefba703d3aacb31cc3f3b2a06f11e454ad
  Stored in directory: /root/.cache/pip/wheels/a2/19/da/a15d4e2bedf3062c739b190d5cb5b7b2ecfbccb6b0d93c861b
Successfully built distlib
Installing collected packages: appdirs, filelock, six, distlib, zipp, importlib-metadata, virtualenv
Successfully installed appdirs-1.4.3 distlib-0.3.0 filelock-3.0.12 importlib-metadata-1.5.0 six-1.14.0 virtualenv-20.0.6 zipp-3.0.0
root@6fad6b6df528:/# virtualenv x
created virtual environment CPython3.7.6.final.0-64 in 418ms
  creator CPython3Posix(dest=/x, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/tmp/tmp2x7cqktt/seed-app-data/v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
root@6fad6b6df528:/# source x/bin/activate
(x) root@6fad6b6df528:/# pip install black
Collecting black
  Downloading black-19.10b0-py36-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 1.8 MB/s 
Collecting regex
  Downloading regex-2020.2.20-cp37-cp37m-manylinux2010_x86_64.whl (689 kB)
     |████████████████████████████████| 689 kB 8.5 MB/s 
Collecting toml>=0.9.4
  Downloading toml-0.10.0-py2.py3-none-any.whl (25 kB)
Collecting typed-ast>=1.4.0
  Downloading typed_ast-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (737 kB)
     |████████████████████████████████| 737 kB 10.2 MB/s 
Collecting pathspec<1,>=0.6
  Downloading pathspec-0.7.0-py2.py3-none-any.whl (25 kB)
Collecting click>=6.5
  Downloading Click-7.0-py2.py3-none-any.whl (81 kB)
     |████████████████████████████████| 81 kB 5.0 MB/s 
Collecting appdirs
  Using cached appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Collecting attrs>=18.1.0
  Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Installing collected packages: regex, toml, typed-ast, pathspec, click, appdirs, attrs, black
Successfully installed appdirs-1.4.3 attrs-19.3.0 black-19.10b0 click-7.0 pathspec-0.7.0 regex-2020.2.20 toml-0.10.0 typed-ast-1.4.1

[La0]

This is upstream bug pypa/pip#7778

Glad you're aware of the issue. So if i understand correctly this comment, this is broken on purpose, and need to wait for a fix on pip itself ?

[gaborbernat]

Broken on purpose, I don't think so, but it's a bug within pip isolated build logic that has been there since it was implemented.

[gaborbernat]

Keeping this issue open, in case we might need to stop generating the pth file until pip fixes that issue and does a release (a workaround for the problem not manifesting as upfront).

[gaborbernat]

We've reverted for now the pth file generation https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-7-2020-02-26

[ofek]

@gaborbernat We had to go back to 20.0.5 b/c it looks like pkg_resources.working_set no longer shows editable installs on Python 2

[gaborbernat]

That should be the case with virtualenv 20.0.5 too 🤔I'm amazed it's not 😲😲😲

[gaborbernat]

Can you post a reproduction of that?

[ofek]

git clone https://github.com/DataDog/integrations-core
python3 -m pip install -e "./integrations-core/datadog_checks_dev[cli]"
python3 -m pip install virtualenv==20.0.6
ddev config set core ./integrations-core
ddev test datadog_checks_base:py27 -k test_utils_agent_packages

[ofek]

Then wipe datadog_checks_base/.tox/py27 & get virtualenv==20.0.7, it will also fail

[ofek]

Did you reproduce?

[gaborbernat]

Not yet and will be unavailable to troubleshoot until next Tuesday now. Sorry I'm on holiday now.

[gaborbernat]

This has been now released again with 20.0.8 and should be fixed. Let me know if not.