Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When $TMPDIR is within $PWD, pip wheel dies on infinite recursion #7872

Closed
hroncok opened this issue Mar 19, 2020 · 2 comments · Fixed by #7873
Closed

When $TMPDIR is within $PWD, pip wheel dies on infinite recursion #7872

hroncok opened this issue Mar 19, 2020 · 2 comments · Fixed by #7873
Labels
auto-locked Outdated issues that have been locked by automation

Comments

@hroncok
Copy link
Contributor

hroncok commented Mar 19, 2020

Environment

  • pip version: master, 20.0.2, 19.3.1... possibly more
  • Python version: 3.9.0a4, 3.8.2, 3.7.7... possibly all
  • OS: Linux (Fedora)

Description

During debugging the problem described in #7555 (comment) we ahve realized that simply setting $TMPDIR to $PWD/.tmp would make our problems go away. However, when doing that, pip wheel tries to copy the entire $PWD to $TMPDIR/pip-req-build-... and it dies out on recursion:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/usr/lib/python3.8/site-packages/pip/_internal/commands/wheel.py", line 161, in run
    resolver.resolve(requirement_set)
  File "/usr/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 178, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/usr/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 336, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 285, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 486, in prepare_linked_requirement
    local_path = unpack_url(
  File "/usr/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 284, in unpack_url
    return unpack_file_url(link, location, download_dir, hashes=hashes)
  File "/usr/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 230, in unpack_file_url
    _copy_source_tree(link_path, location)
  File "/usr/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 213, in _copy_source_tree
    shutil.copytree(source, target, **kwargs)
  File "/usr/lib64/python3.8/shutil.py", line 554, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/usr/lib64/python3.8/shutil.py", line 510, in _copytree
    raise Error(errors)
shutil.Error: [('/builddir/build/BUILD/mistune-0.8.3/.../.tmp/pip-req-build-0pdab7qx/.tmp/pip-req-build-0pdab7qx/__pycache__/mistune.cpython-38.pyc', '/builddir/build/BUILD/mistune-0.8.3/.../.tmp/pip-req-build-0pdab7qx/.tmp/pip-req-build-0pdab7qx/.tmp/pip-req-build-0pdab7qx/__pycache__/mistune.cpython-38.pyc', "[Errno 36] File name too long: '/builddir/build/BUILD/mistune-0.8.3/.../.tmp/pip-req-build-0pdab7qx/.tmp/pip-req-build-0pdab7qx/.tmp/pip-req-build-0pdab7qx/__pycache__/mistune.cpython-38.pyc'"), ...]

Expected behavior

The target directory itself should be ignored when copying to it.

How to Reproduce

$ export TMPDIR=${PWD}/.tmp
$ mkdir ${TMPDIR}
$ pip wheel .  # with an extension module

Output

+ /usr/bin/python3 -m pip wheel --wheel-dir ./pyproject-macros-wheeldir --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --no-clean --progress-bar off --verbose .
Created temporary directory: /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-ephem-wheel-cache-40ziqhu0
Created temporary directory: /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-tracker-kg6rp8lh
Initialized build tracking at /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-tracker-kg6rp8lh
Created build tracker: /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-tracker-kg6rp8lh
Entered build tracker: /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-tracker-kg6rp8lh
Created temporary directory: /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-wheel-d6bw8m93
Processing /builddir/build/BUILD/mistune-0.8.3
  Created temporary directory: /builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-build-0pdab7qx
Removed build tracker: '/builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-tracker-kg6rp8lh'
ERROR: Exception:
(see above)
@triage-new-issues triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label Mar 19, 2020
@hroncok
Copy link
Contributor Author

hroncok commented Mar 19, 2020

I have a fix ready and will submit a Pull Request shorty.

@hroncok
Copy link
Contributor Author

hroncok commented Mar 19, 2020

#7873

hroncok added a commit to hroncok/pip that referenced this issue Mar 19, 2020
During a build of extension module within `pip wheel` the source directory is
recursively copied in a temporary directory.

See pypa#7555

When the temporary directory is inside the source directory
(for example by setting `TMPDIR=$PWD/tmp`) this caused an infinite recursion
that ended in:

    [Errno 36] File name too long

We prevent that buy never copying the target to the target in _copy_source_tree.

Fixes pypa#7872
bors bot referenced this issue in duckinator/emanate May 13, 2020
118: Update pip to 20.1 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.0.2** to **20.1**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.1
   ```
   =================

Process
-------

- Document that pip 21.0 will drop support for Python 2.7.

Features
--------

- Add ``pip cache dir`` to show the cache directory. (`7350 &lt;https://github.com/pypa/pip/issues/7350&gt;`_)

Bug Fixes
---------

- Abort pip cache commands early when cache is disabled. (`8124 &lt;https://github.com/pypa/pip/issues/8124&gt;`_)
- Correctly set permissions on metadata files during wheel installation,
  to permit non-privileged users to read from system site-packages. (`8139 &lt;https://github.com/pypa/pip/issues/8139&gt;`_)
   ```
   
  
  
   ### 20.1b1
   ```
   ===================

Deprecations and Removals
-------------------------

- Remove emails from AUTHORS.txt to prevent usage for spamming, and only populate names in AUTHORS.txt at time of release (`5979 &lt;https://github.com/pypa/pip/issues/5979&gt;`_)
- Remove deprecated ``--skip-requirements-regex`` option. (`7297 &lt;https://github.com/pypa/pip/issues/7297&gt;`_)
- Building of local directories is now done in place, instead of a temporary
  location containing a copy of the directory tree. (`7555 &lt;https://github.com/pypa/pip/issues/7555&gt;`_)
- Remove unused ``tests/scripts/test_all_pip.py`` test script and the ``tests/scripts`` folder. (`7680 &lt;https://github.com/pypa/pip/issues/7680&gt;`_)

Features
--------

- pip now implements PEP 610, so ``pip freeze`` has better fidelity
  in presence of distributions installed from Direct URL requirements. (`609 &lt;https://github.com/pypa/pip/issues/609&gt;`_)
- Add ``pip cache`` command for inspecting/managing pip&#39;s wheel cache. (`6391 &lt;https://github.com/pypa/pip/issues/6391&gt;`_)
- Raise error if ``--user`` and ``--target`` are used together in ``pip install`` (`7249 &lt;https://github.com/pypa/pip/issues/7249&gt;`_)
- Significantly improve performance when ``--find-links`` points to a very large HTML page. (`7729 &lt;https://github.com/pypa/pip/issues/7729&gt;`_)
- Indicate when wheel building is skipped, due to lack of the ``wheel`` package. (`7768 &lt;https://github.com/pypa/pip/issues/7768&gt;`_)
- Change default behaviour to always cache responses from trusted-host source. (`7847 &lt;https://github.com/pypa/pip/issues/7847&gt;`_)
- An alpha version of a new resolver is available via ``--unstable-feature=resolver``. (`988 &lt;https://github.com/pypa/pip/issues/988&gt;`_)

Bug Fixes
---------

- Correctly freeze a VCS editable package when it is nested inside another VCS repository. (`3988 &lt;https://github.com/pypa/pip/issues/3988&gt;`_)
- Correctly handle ``%2F`` in URL parameters to avoid accidentally unescape them
  into ``/``. (`6446 &lt;https://github.com/pypa/pip/issues/6446&gt;`_)
- Reject VCS URLs with an empty revision. (`7402 &lt;https://github.com/pypa/pip/issues/7402&gt;`_)
- Warn when an invalid URL is passed with ``--index-url`` (`7430 &lt;https://github.com/pypa/pip/issues/7430&gt;`_)
- Use better mechanism for handling temporary files, when recording metadata
  about installed files (RECORD) and the installer (INSTALLER). (`7699 &lt;https://github.com/pypa/pip/issues/7699&gt;`_)
- Correctly detect global site-packages availability of virtual environments
  created by PyPA’s virtualenv&gt;=20.0. (`7718 &lt;https://github.com/pypa/pip/issues/7718&gt;`_)
- Remove current directory from ``sys.path`` when invoked as ``python -m pip &lt;command&gt;`` (`7731 &lt;https://github.com/pypa/pip/issues/7731&gt;`_)
- Stop failing uninstallation, when trying to remove non-existent files. (`7856 &lt;https://github.com/pypa/pip/issues/7856&gt;`_)
- Prevent an infinite recursion with ``pip wheel`` when ``$TMPDIR`` is within the source directory. (`7872 &lt;https://github.com/pypa/pip/issues/7872&gt;`_)
- Significantly speedup ``pip list --outdated`` by parallelizing index interaction. (`7962 &lt;https://github.com/pypa/pip/issues/7962&gt;`_)
- Improve Windows compatibility when detecting writability in folder. (`8013 &lt;https://github.com/pypa/pip/issues/8013&gt;`_)

Vendored Libraries
------------------

- Update semi-supported debundling script to reflect that appdirs is vendored.
- Add ResolveLib as a vendored dependency.
- Upgrade certifi to 2020.04.05.1
- Upgrade contextlib2 to 0.6.0.post1
- Upgrade distro to 1.5.0.
- Upgrade idna to 2.9.
- Upgrade msgpack to 1.0.0.
- Upgrade packaging to 20.3.
- Upgrade pep517 to 0.8.2.
- Upgrade pyparsing to 2.4.7.
- Remove pytoml as a vendored dependency.
- Upgrade requests to 2.23.0.
- Add toml as a vendored dependency.
- Upgrade urllib3 to 1.25.8.

Improved Documentation
----------------------

- Emphasize that VCS URLs using git, git+git and git+http are insecure due to
  lack of authentication and encryption (`1983 &lt;https://github.com/pypa/pip/issues/1983&gt;`_)
- Clarify the usage of --no-binary command. (`3191 &lt;https://github.com/pypa/pip/issues/3191&gt;`_)
- Clarify the usage of freeze command in the example of Using pip in your program (`7008 &lt;https://github.com/pypa/pip/issues/7008&gt;`_)
- Add a &quot;Copyright&quot; page. (`7767 &lt;https://github.com/pypa/pip/issues/7767&gt;`_)
- Added example of defining multiple values for options which support them (`7803 &lt;https://github.com/pypa/pip/issues/7803&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <[email protected]>
@lock lock bot added the auto-locked Outdated issues that have been locked by automation label May 20, 2020
@triage-new-issues triage-new-issues bot removed the S: needs triage Issues/PRs that need to be triaged label May 20, 2020
@lock lock bot locked as resolved and limited conversation to collaborators May 20, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked Outdated issues that have been locked by automation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant