Pip called by poetry tries to uninstall system site-packages

[JanLuca]

• Poetry version: 1.3.1
• Python version: 3.10.6
• OS version and name: Debian unstable
• pyproject.toml:

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

The pip command with the arguments provided by poetry wants to uninstall system site-packages. The pyproject.toml is just a new generated one with pylint in the newest version as entry.

$ poetry install -vvv
Loading configuration file /home/jan/.config/pypoetry/config.toml
Creating virtualenv test-CHLv_kh8-py3.10 in /home/jan/.local/share/pypoetry/virtualenvs
Using virtualenv: /home/jan/.local/share/pypoetry/virtualenvs/test-CHLv_kh8-py3.10
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 0 installs, 10 updates, 0 removals

  • Updating lazy-object-proxy (1.7.1 /usr/lib/python3/dist-packages -> 1.8.0): Pending...
  • Updating lazy-object-proxy (1.7.1 /usr/lib/python3/dist-packages -> 1.8.0): Installing...
  • Updating lazy-object-proxy (1.7.1 /usr/lib/python3/dist-packages -> 1.8.0): Failed

  Stack trace:

  2  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1536 in _run
      1534│                 )
      1535│             else:
    → 1536│                 output = subprocess.check_output(
      1537│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1538│                 )

  1  /usr/lib/python3.10/subprocess.py:421 in check_output
       419│         kwargs['input'] = empty
       420│ 
    →  421│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
       422│                **kwargs).stdout
       423│ 

  CalledProcessError

  Command '['python', '/home/jan/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-22.3.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/home/jan/.local/share/pypoetry/virtualenvs/test-CHLv_kh8-py3.10', '--upgrade', '--no-deps', '/home/jan/.cache/pypoetry/artifacts/63/98/c6/19649371fc5357775e3397b1177bfd13829d8fb6408e296076b8ff63b8/lazy-object-proxy-1.8.0.tar.gz']' returned non-zero exit status 1.

  at /usr/lib/python3.10/subprocess.py:526 in run
       522│             # We don't call process.wait() as .__exit__ does that for us.
       523│             raise
       524│         retcode = process.poll()
       525│         if check and retcode:
    →  526│             raise CalledProcessError(retcode, process.args,
       527│                                      output=stdout, stderr=stderr)
       528│     return CompletedProcess(process.args, retcode, stdout, stderr)
       529│ 
       530│ 

The following error occurred when trying to handle this error:


  Stack trace:

  3  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/pip.py:56 in pip_install
       54│ 
       55│     try:
    →  56│         return environment.run_pip(*args)
       57│     except EnvCommandError as e:
       58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e

  2  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1497 in run_pip
      1495│         pip = self.get_pip_command()
      1496│         cmd = pip + list(args)
    → 1497│         return self._run(cmd, **kwargs)
      1498│ 
      1499│     def run_python_script(self, content: str, **kwargs: Any) -> int | str:

  1  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1776 in _run
      1774│     def _run(self, cmd: list[str], **kwargs: Any) -> int | str:
      1775│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
    → 1776│         return super()._run(cmd, **kwargs)
      1777│ 
      1778│     def get_temp_environ(

  EnvCommandError

  Command ['python', '/home/jan/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-22.3.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/home/jan/.local/share/pypoetry/virtualenvs/test-CHLv_kh8-py3.10', '--upgrade', '--no-deps', '/home/jan/.cache/pypoetry/artifacts/63/98/c6/19649371fc5357775e3397b1177bfd13829d8fb6408e296076b8ff63b8/lazy-object-proxy-1.8.0.tar.gz'] errored with the following return code 1, and output: 
  Processing /home/jan/.cache/pypoetry/artifacts/63/98/c6/19649371fc5357775e3397b1177bfd13829d8fb6408e296076b8ff63b8/lazy-object-proxy-1.8.0.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Building wheels for collected packages: lazy-object-proxy
    Building wheel for lazy-object-proxy (pyproject.toml): started
    Building wheel for lazy-object-proxy (pyproject.toml): finished with status 'done'
    Created wheel for lazy-object-proxy: filename=lazy_object_proxy-1.8.0-cp310-cp310-linux_x86_64.whl size=58729 sha256=9cbdbbb91b1ba143a23733920631160cb2a4519fbff4ad05aa79d244c6ea370c
    Stored in directory: /run/user/1000/cache/pip/wheels/7b/13/96/1a0e4c462677931f4fdc17caad514aab94dc011a41f2c95119
  Successfully built lazy-object-proxy
  Installing collected packages: lazy-object-proxy
    Attempting uninstall: lazy-object-proxy
      Found existing installation: lazy-object-proxy 1.7.1
      Uninstalling lazy-object-proxy-1.7.1:
  ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'slots.py'
  Consider using the `--user` option or check the permissions.
  
  

  at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1540 in _run
      1536│                 output = subprocess.check_output(
      1537│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1538│                 )
      1539│         except CalledProcessError as e:
    → 1540│             raise EnvCommandError(e, input=input_)
      1541│ 
      1542│         return decode(output)
      1543│ 
      1544│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  Stack trace:

  6  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:263 in _execute_operation
      261│ 
      262│             try:
    → 263│                 result = self._do_execute_operation(operation)
      264│             except EnvCommandError as e:
      265│                 if e.e.returncode == -2:

  5  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:334 in _do_execute_operation
      332│             return 0
      333│ 
    → 334│         result: int = getattr(self, f"_execute_{method}")(operation)
      335│ 
      336│         if result != 0:

  4  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:461 in _execute_update
      459│ 
      460│     def _execute_update(self, operation: Install | Update) -> int:
    → 461│         status_code = self._update(operation)
      462│ 
      463│         self._save_url_reference(operation)

  3  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:499 in _update
      497│ 
      498│     def _update(self, operation: Install | Update) -> int:
    → 499│         return self._install(operation)
      500│ 
      501│     def _remove(self, operation: Uninstall) -> int:

  2  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:496 in _install
      494│         )
      495│         self._write(operation, message)
    → 496│         return self.pip_install(archive, upgrade=operation.job_type == "update")
      497│ 
      498│     def _update(self, operation: Install | Update) -> int:

  1  ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:125 in pip_install
      123│     ) -> int:
      124│         try:
    → 125│             pip_install(req, self._env, upgrade=upgrade, editable=editable)
      126│         except EnvCommandError as e:
      127│             output = decode(e.e.output)

  PoetryException

  Failed to install /home/jan/.cache/pypoetry/artifacts/63/98/c6/19649371fc5357775e3397b1177bfd13829d8fb6408e296076b8ff63b8/lazy-object-proxy-1.8.0.tar.gz

  at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/pip.py:58 in pip_install
       54│ 
       55│     try:
       56│         return environment.run_pip(*args)
       57│     except EnvCommandError as e:
    →  58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       59│ 

  • Updating wrapt (1.14.1 /usr/lib/python3/dist-packages -> 1.14.1): Pending...
  • Updating wrapt (1.14.1 /usr/lib/python3/dist-packages -> 1.14.1): Installing...
  • Updating wrapt (1.14.1 /usr/lib/python3/dist-packages -> 1.14.1)

[dimbleby]

Use a virtual environment.

Currently you are telling poetry to install packages per the solution that it has found - and then not liking it when it tries to do exactly that.

Edit: or possibly, use a virtual environment that does not share the system site-packages

#6398

[JanLuca]

I am using virtual envs (which is created by poetry) as you can see in the output of the command: ``` Creating virtualenv test-CHLv_kh8-py3.10 in /home/jan/.local/share/pypoetry/virtualenvs Using virtualenv: /home/jan/.local/share/pypoetry/virtualenvs/test-CHLv_kh8-py3.10 ``` The pip command also includes this path as `--prefix` in its call which is the for me confusing part. Am 18. Dezember 2022 14:13:35 MEZ schrieb David Hotham ***@***.***>:

…

Use a virtual environment. Currently you are telling poetry to install packages per the solution that it has found - and then not liking it when it tries to do exactly that.

[dimbleby]

Then, per my edit:

use a virtual environment that does not share the system site-packages

poetry defaults to not sharing system site-packages. If you have configured it to do so, then... don't!

[JanLuca]

I did not change the standard configuration. virtualenvs.options.system-site-packages is set to false so I did not change this behavior. This problem is new after I updated from 1.2.0 to 1.3.1 (without any config change)...

$ poetry config --list
cache-dir = "/home/jan/.cache/pypoetry"
experimental.new-installer = true
experimental.system-git-client = false
installer.max-workers = null
installer.no-binary = null
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "/home/jan/.local/share/pypoetry/virtualenvs"
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"

[dimbleby]

And yet the virtual environment that you are creating is clearly sharing system site-packages. You'll need to figure out why that's happening for you and not for anyone else.

Perhaps you have a virtualenv.ini somewhere that is being picked up?

[JanLuca]

The thing is: If I create a virtual env without poetry and copy the pip command from the poetry debug output everything works as expected:

$ python3 -m venv foobar
$ . foobar/bin/activate
(foobar) $ python /home/jan/.local/share/pypoetry/venv/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-22.3.1-py3-none-any.whl/pip install install --use-pep517 --disable-pip-version-check --isolated --no-input --upgrade --no-deps /home/jan/.cache/pypoetry/artifacts/63/98/c6/19649371fc5357775e3397b1177bfd13829d8fb6408e296076b8ff63b8/lazy-object-proxy-1.8.0.tar.gz
Processing /home/jan/.cache/pypoetry/artifacts/63/98/c6/19649371fc5357775e3397b1177bfd13829d8fb6408e296076b8ff63b8/lazy-object-proxy-1.8.0.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting install
  Downloading install-1.3.5-py3-none-any.whl (3.2 kB)
Building wheels for collected packages: lazy-object-proxy
  Building wheel for lazy-object-proxy (pyproject.toml) ... done
  Created wheel for lazy-object-proxy: filename=lazy_object_proxy-1.8.0-cp310-cp310-linux_x86_64.whl size=58728 sha256=227df64235851d78d2f98a873c6ec04a4ca838d11673c3c116ca85634ec8ed22
  Stored in directory: /run/user/1000/cache/pip/wheels/7b/13/96/1a0e4c462677931f4fdc17caad514aab94dc011a41f2c95119
Successfully built lazy-object-proxy
Installing collected packages: lazy-object-proxy, install
Successfully installed install-1.3.5 lazy-object-proxy-1.8.0

[dimbleby]

Suggest trying to build a docker image that reproduces the problem

• if you succeed, that'll give poetry maintainers something to look at
• if you fail, that will help you to test what is special about your environment

[JanLuca]

Ok, after a longer debug of virtualenv I found the problem.
In pypa/virtualenv@445a68d they fix upstream the prefix on Debian systems with 3.10. But virtualenv caches the results from their PyInfo class so the fix was not applied on my system since it looks like I called poetry at least once after installing 3.10 but before they fixed this.

If someone else is so unlucky to encounter the problem: Just nuke your virtualenv cache dir which is under .local/share/virtualenv.

[JanLuca]

For completeness: I raised the bug that the cache is not invalidated to upstream: pypa/virtualenv#2467

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.