Cleanup fails on Windows when build dir includes junctions

[vidartf]

• Pip version: 9, 10 (more?)
• Python version: 2.7 - 3.6
• Operating system: Windows

Description:

When running pip install <source install target> which triggers a build step that creates Windows directory junctions, the cleanup step of pip can fail with a FileNotFoundError [WinError 3]. The file(s) in question is within one of the directory junctions in the temporary folder created by pip.

The underlying cause is Python issue 31226, and the issue is explained further there. What I'm hoping for is either:

• a way to work around this problem from the user's side.
• or a workaround implemented in pip's cleanup step.

What I've run:

pip install ., being in a source checkout for a package that will create internal directory junctions during its build.

Stack trace

Traceback (most recent call last):
  File "c:\python35-x64\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "c:\python35-x64\lib\site-packages\pip\req\req_set.py", line 796, in install
    requirement.remove_temporary_source()
  File "c:\python35-x64\lib\site-packages\pip\req\req_install.py", line 977, in remove_temporary_source
    rmtree(self.source_dir)
  File "c:\python35-x64\lib\site-packages\pip\_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "c:\python35-x64\lib\site-packages\pip\_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "c:\python35-x64\lib\site-packages\pip\_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "c:\python35-x64\lib\site-packages\pip\_vendor\six.py", line 686, in reraise
    raise value
  File "c:\python35-x64\lib\site-packages\pip\_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "c:\python35-x64\lib\site-packages\pip\utils\__init__.py", line 102, in rmtree
    onerror=rmtree_errorhandler)
  File "c:\python35-x64\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\python35-x64\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\python35-x64\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\python35-x64\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\python35-x64\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\python35-x64\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\python35-x64\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\python35-x64\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\python35-x64\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\python35-x64\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\python35-x64\lib\shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\python35-x64\lib\site-packages\pip\utils\__init__.py", line 110, in rmtree_errorhandler
    if os.stat(path).st_mode & stat.S_IREAD:
FileNotFoundError: [WinError 3] The system cannot find the path specified: 

[pradyunsg]

Hey @vidartf! Thanks for filing this issue and collapsing the stack trace. :)

Is this equivalent to #5306, on Windows?

[pfmoore]

@pradyunsg Not really, in that #5306 is about a special file in the source directory which can't be copied to the temporary build directory. This is about the build stage creating a special file that pip can't delete when removing the temporary build directory.

A workaround for this issue might be that pip simply ignores errors when trying to delete the temporary build directory. The user would then have to delete it manually (we could maybe write a warning giving the name of the directory that we failed to delete).

[pradyunsg]

Ah. Okhay. Thanks for the clarification @pfmoore. :)

[vidartf]

This is about the build stage creating a special file that pip can't delete when removing the temporary build directory.

Just to clarify: This is about Python not recognizing junction directory links properly, so rmtree tries to delete the content of a folder twice. There are no files that cannot be removed here.

[pfmoore]

@vidartf Thanks for the clarification. Nevertheless, I think the best pip could do is report that it failed to delete the directory and leave it to the user to do so manually.

[vidartf]

@pfmoore Sure, but as I do not understand all the nuances involved, I just wanted to make the case I was reporting clear. Your solution seems good to me!