pip install fails to remove its temporary directory on Windows when it contains a deep folder hierarchy.

[stefano-m]

Hi,

First off, I am using the following:

• WIndows 7
• Python 2.7.9
• pip 7.0.3
• virtualenv 13.0.3
• setuptools 17.0

I will try and publicly upload and link a simple example for you to peruse later today.

TL;DR

Doing pip install on Windows for a package that creates a deep hierarchy of directories (e.g. by using npm install internally) fails to remove its temporary build directory pip-*-build because of the path length limitations imposed by Windows. This leaves the package "half installed".

A workaround I have found is to wrap pip.utils.rmtree in a try: except block that ignores the specific WindowsError related to the path length limitations. This can work as a one-off because it requires to modify pip's own code and I don't feel is completely satisfactory. Also, it does not actually remove the pip-*-build directory that will be left dangling.

Long long description

I have a Python package for a web app that uses npm to download its JavaScript requirements. To do so, I have created a custom install class that runs npm install with subprocess before running the actual installation run method and have overridden the install class in my setup function using the cmdclass keyword argument.

pip install works fine on GNU/Linux, but when I try it on Windows 7, it fails to remove the temporary directory it created and lefts the package half installed. After some Internet searching, I think this is due to the Maximum Path Length Limitation imposed on Windows and the way npm stores its dependencies in a nested way.

The error reported is

File "C:\Python27\Lib\shutil.py", line 252, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "f:\sandbox_venv\lib\site-packages\pip\utils\__init__.py", line 97, in rmtree_errorhandler
    if os.stat(path).st_mode & stat.S_IREAD:
WindowsError: [Error 3] The system cannot find the path specified: 'c:\\users\\testuser\\appdata\\local\\temp\\pip-xdfobz-build\\node_modules\\bower\\node_modules\\update-notifier\\node_modules\\latest-version\\node_modules\\package-json\\node_modules\\registry-url\\node_modules\\npmconf\\node_modules\\config-chain\\node_modules\\proto-list\\LICENSE

The failure is in the rmtree_errorhandler function in pip\utils\__init__.py, where it does an os.stat(path). I tried to prepend the path with the \\\\?\\ string as suggested in this stackoverflow question by adding something like the following:

if sys.platform == 'win32':
    path = '\\\\?\\' + path

• If I do that in pip.utils.rmtree, I get

  • for pip install:

  File "f:\sandbox_venv\lib\site-packages\pip\utils\__init__.py", line 92, in rmtree
    onerror=rmtree_errorhandler)
  File "C:\Python27\Lib\shutil.py", line 239, in rmtree
    onerror(os.listdir, path, sys.exc_info())
  File "f:\sandbox_venv\lib\site-packages\pip\utils\__init__.py", line 108, in rmtree_errorhandler
    func(path)
  WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: '\\\\?\\c:\\users\\testuser\\appdata\\local\\temp\\pip-build-_v2szh/*.*'

  • for pip uninstall (if trying to remove an half-installed package):

  File "C:\Python27\Lib\shutil.py", line 239, in rmtree
  onerror(os.listdir, path, sys.exc_info())
  File "f:\sandbox_venv\lib\site-packages\pip\utils\__init__.py", line 104, in rmtree_errorhandler
  func(path)
  WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: '\\\\?\\c:\\users\\testuser\\appdata\\local\\temp\\pip-1nz71w-uninstall/*.*'

• If I do that in pip.utils.rmtree_errorhandler, I get

  File "C:\Python27\Lib\shutil.py", line 239, in rmtree
    onerror(os.listdir, path, sys.exc_info())
  File "f:\sandbox_venv\lib\site-packages\pip\utils\__init__.py", line 104, in rmtree_errorhandler
    func(path)
TypeError: must be (buffer overflow), not str

• If I do that in pip.utils.rmtree_errorhandler, but only for the path variable in os.stat, I get and error on the following os.chmod:

  File "f:\sandbox_venv\lib\site-packages\pip\utils\__init__.py", line 105, in rmtree_errorhandler
    os.chmod(path, stat.S_IWRITE)
WindowsError: [Error 3] The system cannot find the path specified: 'c:\\users\\testuser\\appdata\\local\\temp\\pip-chyhk8-build\\node_modules\\bower\\node_modules\\update-notifier\\node_modules\\latest-version\\node_modules\\package-json\\node_modules\\registry-url\\node_modules\\npmconf\\node_modules\\config-chain\\node_modules\\proto-list\\LICENSE'

• If I do that in pip.utils.rmtree_errorhandler, but only for the path variable in both os.stat and os.chmod, I get and error on the following func:

  File "C:\Python27\Lib\shutil.py", line 252, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "f:\sandbox_venv\lib\site-packages\pip\utils\__init__.py", line 107, in rmtree_errorhandler
    func(path)
WindowsError: [Error 3] The system cannot find the path specified: 'c:\\users\\testuser\\appdata\\local\\temp\\pip-rhukyx-build\\node_modules\\bower\\node_modules\\update-notifier\\node_modules\\latest-version\\node_modules\\package-json\\node_modules\\registry-url\\node_modules\\npmconf\\node_modules\\config-chain\\node_modules\\proto-list\\LICENSE'

Also, if I wrap the code of pip.utils.rmtree_errorhandler around a try: except: that will return only if WindowsError.winerror == 3 I get:

  File "C:\Python27\Lib\shutil.py", line 256, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "f:\sandbox_venv\lib\site-packages\pip\utils\__init__.py", line 102, in rmtree_errorhandler
    func(path)
WindowsError: [Error 145] The directory is not empty: 'c:\\users\\testuser\\appdata\\local\\temp\\pip-hyti3u-build\\node_modules\\bower\\node_modules\\update-notifier\\node_modules\\latest-version\\node_modules\\package-json\\node_modules\\registry-url\\node_modules\\npmconf\\node_modules\\config-chain\\node_modules\\proto-list\\test'

Finally, I could get around the error by using the try: except: in pip.utils.rmtree:

# Retry every half second for up to 3 seconds
@retry(stop_max_delay=3000, wait_fixed=500)
def rmtree(dir, ignore_errors=False):
    try:
        shutil.rmtree(dir, ignore_errors=ignore_errors,
                      onerror=rmtree_errorhandler)
    except WindowsError as e:
        if e.winerror == 3:
            return
        raise

but of course, that does not actually remove the temporary directory pip-*-build.

[stefano-m]

Here is my working example:

https://github.com/stefano-m/pip_issue_2892

[mindw]

Does the patch below can addresses this issue?
mindw/pip@5ca7e2a

[stefano-m]

Gabi Davar [email protected] writes:

Does the patch below can addresses this isssue?
mindw/pip@5ca7e2a

—
Reply to this email directly or view it on GitHub.

Hi,

thanks for the patch. I will try this out during the week (when I have
access to a Windows machine).

Btw, there's a typo in the comment in your patch, litetal instead of
literal and you wrote bypass twice ;)

-- Stefano

[stefano-m]

@mindw , unfortunately the patch is not sufficient.
pip still falls over in rmtree_errorhandler when it calls os.stat. It will likely also fall over when calling os.chmod in the same function.

[mindw]

Thanks for testing this - I'll try and revise the patch.

-gabi

On Mon, Jun 15, 2015 at 12:40 PM, Stefano Mazzucco <[email protected]

wrote:

@mindw https://github.com/mindw , unfortunately the patch is not
sufficient.
pip still falls over in rmtree_errorhandler when it calls os.stat. It
will likely also fall over when calling os.chmod in the same function.

—
Reply to this email directly or view it on GitHub
#2892 (comment).

[stefano-m]

@mindw If you have access to a Windows machine you can try out my example

https://github.com/stefano-m/pip_issue_2892

(you will need node.js too).

[stefano-m]

First of all

Even though I have a workaround, I am happy to test possible fixes for pip. My workaround applies only if the problem is caused by npm. So it would be great if pip managed to clean up long paths on Windows in all cases.

Workaround

I have a workaround that can help me solve the issue: I run npm uninstall <NODE_MODULE> as cleanup action after I am done with the node-dependent tasks.

Now my install and build command classes do something like:

1. run npm install
2. run the node-dependent tasks (e.g. gulp to compile less to css)
3. if on Windows: loop through the modules in node_modules and run npm uninstall <NODE_MODULE
4. go on with the usual install or build process.

By letting npm cleaning up after itself, I can make pip happy to remove its temporary directory.

I have pushed the workaround in a branch of my working example:

https://github.com/stefano-m/pip_issue_2892/tree/workaround

[stefano-m]

I was thinking of adding a try/catch around the part the generates the problem and log a warning about the fact that the temporary directory could not be deleted.

However I am not sure whether this may affect other errors not related to long paths, so I'd appreciate some advice.

[kevhill]

For what it is worth, I'm also seeing this error on a Docker container based on debian:latest

pip: 7.1.2

    Uninstalling six-1.8.0:
      Successfully uninstalled six-1.8.0
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 211, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 311, in run
    root=options.root_path,
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 657, in install
    requirement.commit_uninstall()
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 729, in commit_uninstall
    self.uninstalled.commit()
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_uninstall.py", line 152, in commit
    rmtree(self.save_dir)
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/retrying.py", line 212, in call
    raise attempt.get()
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/local/lib/python2.7/dist-packages/pip/utils/__init__.py", line 90, in rmtree
    onerror=rmtree_errorhandler)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 252, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "/usr/local/lib/python2.7/dist-packages/pip/utils/__init__.py", line 98, in rmtree_errorhandler
    if os.stat(path).st_mode & stat.S_IREAD:
OSError: [Errno 2] No such file or directory: '/tmp/pip-zZsd__-uninstall/usr/lib/python2.7/dist-packages/six.py'

Craziest part is it doesn't error if you just do docker run debian bash and put the same lines in from an interactive terminal

A try-catch might be great.

[stefano-m]

@kevhill, I agree with you that the try/catch would be a good deal. I would log the fact that pip couldn't remove the temporary directory and leave that to the user.

What I am worried about is potential side effects. I am not familiar enough with pip's codebase, that's why I have not created a PR.

Guidance from pip's core devs would be great.

[dstufft]

Closing this as a duplicate of #3055.