Existing editable installs do not satisfy dependencies when using 2020-resolver

[alanbriolat]

What did you want to do?

I wanted to check that the new resolver in upcoming pip 20.3 doesn't break my project which is based on Python 2.7 and makes use of several local editable dependencies. I found that the same issues I reported for the old resolver in #8715 (fixed by #8702) are present when using --use-feature=2020-resolver.

A walkthrough of a minimal example is included below. In summary:

• pip version: 20.2.3 and also latest master (ccc84a6)
• Python version: 2.7.17
• OS: Ubuntu 18.04 64-bit
• Existing editable installs, i.e. previously installed with pip install --use-feature=2020-resolver -e package_a/, are not visible to the resolver when installing other packages that depend on them
• This isn't true if supplied at the same time, i.e. pip install --use-feature=2020-resolver -e package_a/ -e package_b/
• (Editable installs also aren't visible to pip uninstall, although I think this is pip uninstall not working with editable dependencies #5193)

Output

$ mkdir -p package_a package_b

$ cat > package_a/setup.py <<EOF
from setuptools import setup, find_packages

setup(
    name="package-a",
    version="0.1",
    packages=find_packages(),
)
EOF

$ cat > package_b/setup.py <<EOF
from setuptools import setup, find_packages

setup(
    name="package-b",
    version="0.1",
    packages=find_packages(),
    install_requires=["package-a"],
)
EOF

$ virtualenv -p python2.7 venv
Running virtualenv with interpreter /usr/bin/python2.7
Already using interpreter /usr/bin/python2.7
New python executable in /home/alanb/tmp/venv/bin/python2.7
Also creating executable in /home/alanb/tmp/venv/bin/python
Installing setuptools, pip, wheel...
done.

$ source venv/bin/activate

$ python -m pip install -U "pip @ https://github.com/pypa/pip/archive/master.zip"
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting pip@ https://github.com/pypa/pip/archive/master.zip
  Using cached https://github.com/pypa/pip/archive/master.zip
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: pip
  Building wheel for pip (PEP 517) ... done
  Created wheel for pip: filename=pip-20.3.dev0-py2.py3-none-any.whl size=1505022 sha256=6b9ea93d657dc30195139a705bd0af903a01e45fe8d26ad43a3f6871e205d161
  Stored in directory: /tmp/pip-ephem-wheel-cache-BXHvS5/wheels/34/c3/dc/0f36a4816c422a97460dfb4e3fdf2a9bc7843678e7536781fd
Successfully built pip
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
Successfully installed pip-20.3.dev0

$ pip --version
pip 20.3.dev0 from /tmp/venv/local/lib/python2.7/site-packages/pip (python 2.7)

$ pip install --use-feature=2020-resolver -e package_a/
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Obtaining file:///tmp/package_a
Installing collected packages: package-a
  Running setup.py develop for package-a
Successfully installed package-a

$ pip install --use-feature=2020-resolver -e package_b/
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Obtaining file:///tmp/package_b
ERROR: Could not find a version that satisfies the requirement package-a (from package-b)
ERROR: No matching distribution found for package-a

$ pip freeze
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
# Editable install with no version control (package-a==0.1)
-e /tmp/package_a

$ pip install --use-feature=2020-resolver -e package_a/ -e package_b/
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Obtaining file:///tmp/package_a
Obtaining file:///tmp/package_b
Installing collected packages: package-a, package-b
  Running setup.py develop for package-a
  Running setup.py develop for package-b
Successfully installed package-a package-b

$ pip freeze
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
# Editable install with no version control (package-a==0.1)
-e /tmp/package_a
# Editable install with no version control (package-b==0.1)
-e /tmp/package_b

$ pip install --use-feature=2020-resolver -e package_b/
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Obtaining file:///tmp/package_b
ERROR: Could not find a version that satisfies the requirement package-a (from package-b)
ERROR: No matching distribution found for package-a

$ pip install -e package_b/
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Obtaining file:///tmp/package_b
Requirement already satisfied: package-a in ./package_a (from package-b==0.1) (0.1)
Installing collected packages: package-b
  Running setup.py develop for package-b
Successfully installed package-b

$ pip uninstall package-a package-b
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Found existing installation: package-b 0.1
Not uninstalling package-b at /tmp/package_b, outside environment /tmp/venv
Can't uninstall 'package-b'. No files were found to uninstall.
Found existing installation: package-a 0.1
Not uninstalling package-a at /tmp/package_a, outside environment /tmp/venv
Can't uninstall 'package-a'. No files were found to uninstall.

$ pip install pipdeptree

$ pipdeptree
package-b==0.1
  - package-a [required: Any, installed: 0.1]
pipdeptree==1.0.0
  - pip [required: >=6.0.0, installed: 20.3.dev0]
setuptools==44.1.1
wheel==0.35.1

Additional information

I'm concerned that pushing to make the new resolver default before 21.0, combined with maintainers who "don't care" about Python 2 support (understandably so, it's end-of-life), is going to leave the Python 2 ecosystem in a worse state than before. In my opinion, it seems unnecessary to risk permanent regressions in the final Python 2 compatible release, when the feature could remain optional.

[uranusjr]

It seems like this is the same as #8785?

[alanbriolat]

I don't know enough about the internals to know if they're related. I did find that issue before creating this one, but it seemed like it was tied up in far more complex interactions with the "extras" functionality, rather than just "editable installs are ignored entirely". Specifically, that issue is a failure even when the dependencies are provided at the same time, rather than resolving already-installed dependencies from the environment, whereas this is more related to the failure to resolve from the environment.

[uranusjr]

Merging this into #8785 to avoid messages from scattering.