AssertionError: Egg-link <package> does not match .. when using symbolic links

[fohrloop]

Environment

• pip version: 20.2.4 and 20.3.3
• Python version: 3.8.6 64-bit
• OS: Windows 10

Description

Install a package in editable state using a path that contains a symbolic link. After that, you can not uninstall the package. This seems to be very much like #6900 (especially: #6900 (comment)). Could be related to #8598.

Expected behavior

Should be able to install, uninstall, reinstall packages to any folder, even if the file path has symbolic links.

How to Reproduce

1) Create project

• Project root: C:\tmp\a_long_long_filepath

Files:

# setup.py
import setuptools

setuptools.setup(name="mypackage")

# mypackage.py 
print("Hello from mypackage")

2) Create symbolic link

• Windows Powershell (Admin)

New-Item  -ItemType SymbolicLink -Path D:\foo -Value C:\tmp\a_long_long_filepath

This creates symbolic link D:\foo that points to C:\tmp\a_long_long_filepath

3) Install the package in editable state

PS C:\Users\USER> pip install -e D:\foo\mypackage\
Obtaining file:///D:/foo/mypackage
Installing collected packages: mypackage
  Running setup.py develop for mypackage
Successfully installed mypackage
WARNING: You are using pip version 20.2.4; however, version 20.3.3 is available.
You should consider upgrading via the 'c:\python\python 3.8.6-64\python.exe -m pip install --upgrade pip' command.
PS C:\Users\USER>

4) Check pip list

mypackage                     0.0.0      c:\tmp\a_long_long_filepath\mypackage

5) Try to uninstall

PS C:\Users\USER> pip uninstall mypackage
Found existing installation: mypackage 0.0.0
ERROR: Exception:
Traceback (most recent call last):
  File "c:\python\python 3.8.6-64\lib\site-packages\pip\_internal\cli\base_command.py", line 228, in _main
    status = self.run(options, args)
  File "c:\python\python 3.8.6-64\lib\site-packages\pip\_internal\commands\uninstall.py", line 89, in run
    uninstall_pathset = req.uninstall(
  File "c:\python\python 3.8.6-64\lib\site-packages\pip\_internal\req\req_install.py", line 685, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "c:\python\python 3.8.6-64\lib\site-packages\pip\_internal\req\req_uninstall.py", line 542, in from_dist
    assert (link_pointer == dist.location), (
AssertionError: Egg-link d:\foo\mypackage does not match installed location of mypackage (at c:\tmp\a_long_long_filepath\mypackage)

Upgrade pip to 20.3.3 and try again -> identical results (with line number changes).

Proposed fix

As we are interested knowing if the file (folder) is the same, compare the target of the symbolic link to the folder:

In _internal\req\req_uninstall.py, instead of

	assert (
		link_pointer == dist.location
	), "Egg-link {} does not match installed location of {} " "(at {})".format(
		link_pointer, dist.project_name, dist.location
	)

use

	assert (
		os.path.realpath(link_pointer) == os.path.realpath(dist.location)
	), "Egg-link {} does not match installed location of {} " "(at {})".format(
		link_pointer, dist.project_name, dist.location
	)

In my case,

In [12]: dist.location
Out[12]: 'c:\\tmp\\a_long_long_filepath\\mypackage'

In [13]: link_pointer
Out[13]: 'd:\\foo\\mypackage'

In [14]: os.path.realpath(link_pointer)
Out[14]: 'C:\\tmp\\a_long_long_filepath\\mypackage'

After the proposed fix:

PS C:\Users\USER> pip uninstall mypackage
Found existing installation: mypackage 0.0.0
Uninstalling mypackage-0.0.0:
  Would remove:
    c:\python\python 3.8.6-64\lib\site-packages\mypackage.egg-link
Proceed (y/n)? y
  Successfully uninstalled mypackage-0.0.0

and there is no entry for mypackage in pip list.

[fohrloop]

Actually, it would be probably better to utilize the normalize_path, that is also used inside the same function as

dist_path = normalize_path(dist.location)

The paths are

In [6]: dist_path
Out[6]: 'c:\\tmp\\a_long_long_filepath\\mypackage'

In [7]: link_pointer
Out[7]: 'd:\\foo\\mypackage'

In [8]: normalize_path(link_pointer)
Out[8]: 'c:\\tmp\\a_long_long_filepath\\mypackage'

Therefore, possible fix is to use

            assert (
                normalize_path(link_pointer) == dist_path
            ), "Egg-link {} does not match installed location of {} " "(at {})".format(
                link_pointer, dist.project_name, dist.location
            )

or even

            with open(develop_egg_link, "r") as fh:
                link_pointer = normalize_path(os.path.normcase(fh.readline().strip()))

            assert (
                link_pointer == dist_path
            ), "Egg-link {} does not match installed location of {} " "(at {})".format(
                link_pointer, dist.project_name, dist.location
            )

as the link_pointer is not used anywhere else.

[fohrloop]

Just to comment, this bug does not only affect uninstalling, but also when trying to reinstall any package that has been installed in editable state. For example, having a package with setup.py:

import setuptools


setuptools.setup(
    name="mypackage",
    version="1.0.0",
    packages=[
        "mypackage",
    ],
)

And then installing it twice with python -m pip install -e . will cause the error.

Example output

PS D:\projects\myproject> python -m venv venv
PS D:\projects\myproject> .\venv\Scripts\activate
(venv) PS D:\projects\myproject> python -m pip install -e .
Obtaining file:///D:/projects/myproject
Installing collected packages: mypackage
  Running setup.py develop for mypackage
Successfully installed mypackage
WARNING: You are using pip version 20.2.1; however, version 21.1.1 is available.
You should consider upgrading via the 'D:\projects\myproject\venv\Scripts\python.exe -m pip install --upgrade pip' command.
(venv) PS D:\projects\myproject> python -m pip install -e .
Obtaining file:///D:/projects/myproject
Installing collected packages: mypackage
  Attempting uninstall: mypackage
    Found existing installation: mypackage 1.0.0
ERROR: Exception:
Traceback (most recent call last):
  File "D:\projects\myproject\venv\lib\site-packages\pip\_internal\cli\base_command.py", line 216, in _main
    status = self.run(options, args)
  File "D:\projects\myproject\venv\lib\site-packages\pip\_internal\cli\req_command.py", line 182, 
in wrapper
    return func(self, options, args)
  File "D:\projects\myproject\venv\lib\site-packages\pip\_internal\commands\install.py", line 412, in run
    installed = install_given_reqs(
  File "D:\projects\myproject\venv\lib\site-packages\pip\_internal\req\__init__.py", line 75, in install_given_reqs
    uninstalled_pathset = requirement.uninstall(
  File "D:\projects\myproject\venv\lib\site-packages\pip\_internal\req\req_install.py", line 684, 
in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "D:\projects\myproject\venv\lib\site-packages\pip\_internal\req\req_uninstall.py", line 542, in from_dist
    assert (link_pointer == dist.location), (
AssertionError: Egg-link d:\projects\myproject does not match installed location of mypackage (at c:\users\nipa\onedrive - vaisala oyj\backup\projects\myproject)

Notes

• The bug only requires that you are working on a folder that is symbolically linked. In this case D:\projects is linked to C:\Some\long\path\where\projects\are.
• Tried also the pip 21.1.1, it did not change the error.

[cr1901]

Was there any forward progress on fixing this issue? I have also run into this as part of a larger installer script that I cannot easily modify to skip the offending packages.

[uranusjr]

This is already fixed.

[cr1901]

@uranusjr Yea, I actually can't duplicate this anymore after removing an old .egg-link, presumably generated from a pip before the issue was fixed.

Ahhh well, this still might be useful for others to know: If you run into trouble w/ a recent pip, try removing old .egg-links from your site-packages. Specifically, .egg-links to --editable packages that were installed from a path containing a symlink.

[piotr-dobrogost]

Searching https://pip.pypa.io/en/stable/news/# for 9452 finds nothing and the only PR linked here (#9453) was closed as bitrotten.
Where was this fixed?

[uranusjr]

It’s #10697, I think we failed to recognise (or just forgot) fixing the junction link use case also fixes symbolic links.