poetry install --sync attempts to remove itself when poetry is installed to a different venv

[novas0x2a]

• Poetry version: 1.4.0, 1.3.2, 1.2.2
• Python version: 3.8.2 (the os-installed python)
• OS version and name: OSX 10.15.7 build 19H2 (catalina, yes, this is a somewhat old version)
• pyproject.toml: See repro script here: https://gist.github.com/novas0x2a/98801bb5e82478f8411d8a0a0362fc53

• I am on the latest stable Poetry version, installed using a recommended method. (advanced/manual mode, but yes)
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I'm seeing a weird problem where poetry attempts to remove itself when poetry install --sync is run. However, unlike other examples of this, poetry is not installed into the venv it's attempting to manage. Here's a link to my repro script.

Here's me running it (with minor redaction):

$ ./repro-poetry.sh test-dir 1.4.0 &>out
$ cat out
+ TESTDIR=test-dir
+ POETRY=1.4.0
+ PYTHON=python3
+ test -e test-dir
+ mkdir -p test-dir
+ setup_poetry test-dir/poetry 1.4.0
+ local venv_path=test-dir/poetry
+ local poetry_version=1.4.0
+ python3 -m venv test-dir/poetry
+ test-dir/poetry/bin/python -m pip install pip==23.0.1 setuptools==67.5.1
Collecting pip==23.0.1
  Using cached https://files.pythonhosted.org/packages/07/51/2c0959c5adf988c44d9e1e0d940f5b074516ecc87e96b1af25f59de9ba38/pip-23.0.1-py3-none-any.whl
Collecting setuptools==67.5.1
  Using cached https://files.pythonhosted.org/packages/99/02/d6ac140d446e939959e80b7e0c55774fa7a22376847ff5fe191674fd1862/setuptools-67.5.1-py3-none-any.whl
Installing collected packages: pip, setuptools
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
  Found existing installation: setuptools 41.2.0
    Uninstalling setuptools-41.2.0:
      Successfully uninstalled setuptools-41.2.0
Successfully installed pip-23.0.1 setuptools-67.5.1
+ test-dir/poetry/bin/python -m pip install poetry==1.4.0
Collecting poetry==1.4.0
  Using cached poetry-1.4.0-py3-none-any.whl (221 kB)
Collecting pyproject-hooks<2.0.0,>=1.0.0
  Using cached pyproject_hooks-1.0.0-py3-none-any.whl (9.3 kB)
Collecting requests<3.0,>=2.18
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting poetry-plugin-export<2.0.0,>=1.3.0
  Using cached poetry_plugin_export-1.3.0-py3-none-any.whl (10 kB)
Collecting trove-classifiers>=2022.5.19
  Using cached trove_classifiers-2023.2.20-py3-none-any.whl (13 kB)
Collecting urllib3<2.0.0,>=1.26.0
  Using cached urllib3-1.26.14-py2.py3-none-any.whl (140 kB)
Collecting packaging>=20.4
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting tomli<3.0.0,>=2.0.1
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting xattr<0.11.0,>=0.10.0
  Using cached xattr-0.10.1-cp38-cp38-macosx_10_9_x86_64.whl (15 kB)
Collecting requests-toolbelt<0.11.0,>=0.9.1
  Using cached requests_toolbelt-0.10.1-py2.py3-none-any.whl (54 kB)
Collecting crashtest<0.5.0,>=0.4.1
  Using cached crashtest-0.4.1-py3-none-any.whl (7.6 kB)
Collecting html5lib<2.0,>=1.0
  Using cached html5lib-1.1-py2.py3-none-any.whl (112 kB)
Collecting cachecontrol[filecache]<0.13.0,>=0.12.9
  Using cached CacheControl-0.12.11-py2.py3-none-any.whl (21 kB)
Collecting dulwich<0.22.0,>=0.21.2
  Using cached dulwich-0.21.3-cp38-cp38-macosx_10_9_x86_64.whl (467 kB)
Collecting poetry-core==1.5.1
  Using cached poetry_core-1.5.1-py3-none-any.whl (465 kB)
Collecting pexpect<5.0.0,>=4.7.0
  Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting shellingham<2.0,>=1.5
  Using cached shellingham-1.5.0.post1-py2.py3-none-any.whl (9.4 kB)
Collecting jsonschema<5.0.0,>=4.10.0
  Using cached jsonschema-4.17.3-py3-none-any.whl (90 kB)
Collecting cleo<3.0.0,>=2.0.0
  Using cached cleo-2.0.1-py3-none-any.whl (77 kB)
Collecting installer<0.7.0,>=0.6.0
  Using cached installer-0.6.0-py3-none-any.whl (452 kB)
Collecting build<0.11.0,>=0.10.0
  Using cached build-0.10.0-py3-none-any.whl (17 kB)
Collecting virtualenv!=20.4.5,!=20.4.6,<21.0.0,>=20.4.3
  Using cached virtualenv-20.20.0-py3-none-any.whl (8.7 MB)
Collecting platformdirs<3.0.0,>=2.5.2
  Using cached platformdirs-2.6.2-py3-none-any.whl (14 kB)
Collecting tomlkit!=0.11.2,!=0.11.3,<1.0.0,>=0.11.1
  Using cached tomlkit-0.11.6-py3-none-any.whl (35 kB)
Collecting filelock<4.0.0,>=3.8.0
  Using cached filelock-3.9.0-py3-none-any.whl (9.7 kB)
Collecting importlib-metadata>=4.4
  Using cached importlib_metadata-6.0.0-py3-none-any.whl (21 kB)
Collecting lockfile<0.13.0,>=0.12.2
  Using cached lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
Collecting pkginfo<2.0.0,>=1.9.4
  Using cached pkginfo-1.9.6-py3-none-any.whl (30 kB)
Collecting keyring<24.0.0,>=23.9.0
  Using cached keyring-23.13.1-py3-none-any.whl (37 kB)
Collecting msgpack>=0.5.2
  Using cached msgpack-1.0.4-cp38-cp38-macosx_10_9_x86_64.whl (73 kB)
Collecting rapidfuzz<3.0.0,>=2.2.0
  Using cached rapidfuzz-2.13.7-cp38-cp38-macosx_10_9_x86_64.whl (1.8 MB)
Collecting webencodings
  Using cached webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting six>=1.9
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting zipp>=0.5
  Using cached zipp-3.15.0-py3-none-any.whl (6.8 kB)
Collecting importlib-resources>=1.4.0
  Using cached importlib_resources-5.12.0-py3-none-any.whl (36 kB)
Collecting pkgutil-resolve-name>=1.3.10
  Using cached pkgutil_resolve_name-1.3.10-py3-none-any.whl (4.7 kB)
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
  Using cached pyrsistent-0.19.3-cp38-cp38-macosx_10_9_universal2.whl (82 kB)
Collecting attrs>=17.4.0
  Using cached attrs-22.2.0-py3-none-any.whl (60 kB)
Collecting jaraco.classes
  Using cached jaraco.classes-3.2.3-py3-none-any.whl (6.0 kB)
Collecting ptyprocess>=0.5
  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.1.0-cp38-cp38-macosx_10_9_x86_64.whl (123 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting distlib<1,>=0.3.6
  Using cached distlib-0.3.6-py2.py3-none-any.whl (468 kB)
Collecting cffi>=1.0
  Using cached cffi-1.15.1-cp38-cp38-macosx_10_9_x86_64.whl (178 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting more-itertools
  Using cached more_itertools-9.1.0-py3-none-any.whl (54 kB)
Installing collected packages: webencodings, trove-classifiers, ptyprocess, msgpack, lockfile, distlib, zipp, urllib3, tomlkit, tomli, six, shellingham, rapidfuzz, pyrsistent, pycparser, poetry-core, platformdirs, pkgutil-resolve-name, pkginfo, pexpect, packaging, more-itertools, installer, idna, filelock, crashtest, charset-normalizer, certifi, attrs, virtualenv, requests, pyproject-hooks, jaraco.classes, importlib-resources, importlib-metadata, html5lib, dulwich, cleo, cffi, xattr, requests-toolbelt, keyring, jsonschema, cachecontrol, build, poetry-plugin-export, poetry
Successfully installed attrs-22.2.0 build-0.10.0 cachecontrol-0.12.11 certifi-2022.12.7 cffi-1.15.1 charset-normalizer-3.1.0 cleo-2.0.1 crashtest-0.4.1 distlib-0.3.6 dulwich-0.21.3 filelock-3.9.0 html5lib-1.1 idna-3.4 importlib-metadata-6.0.0 importlib-resources-5.12.0 installer-0.6.0 jaraco.classes-3.2.3 jsonschema-4.17.3 keyring-23.13.1 lockfile-0.12.2 more-itertools-9.1.0 msgpack-1.0.4 packaging-23.0 pexpect-4.8.0 pkginfo-1.9.6 pkgutil-resolve-name-1.3.10 platformdirs-2.6.2 poetry-1.4.0 poetry-core-1.5.1 poetry-plugin-export-1.3.0 ptyprocess-0.7.0 pycparser-2.21 pyproject-hooks-1.0.0 pyrsistent-0.19.3 rapidfuzz-2.13.7 requests-2.28.2 requests-toolbelt-0.10.1 shellingham-1.5.0.post1 six-1.16.0 tomli-2.0.1 tomlkit-0.11.6 trove-classifiers-2023.2.20 urllib3-1.26.14 virtualenv-20.20.0 webencodings-0.5.1 xattr-0.10.1 zipp-3.15.0
+ setup_project test-dir/project
+ local proj_path=test-dir/project
+ mkdir test-dir/project
+ cat
+ cat
+ cat
+ cd test-dir/project
+ ../poetry/bin/poetry debug info

Poetry
Version: 1.4.0
Python:  3.8.2

Virtualenv
Python:         3.8.2
Implementation: CPython
Path:           NA
Executable:     NA

System
Platform:   darwin
OS:         posix
Python:     3.8.2
Path:       $HOME/test-dir/poetry
Executable: $HOME/test-dir/poetry/bin/python3
+ ../poetry/bin/poetry install -vvv --sync --dry-run
Loading configuration file $HOME/Library/Preferences/pypoetry/config.toml
Loading configuration file $HOME/test-dir/project/poetry.toml
Creating virtualenv test-dir in $HOME/test-dir/project/.venv
Using virtualenv: $HOME/test-dir/project/.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 0 installs, 0 updates, 47 removals

  • Removing attrs (22.2.0)
  • Removing build (0.10.0)
  • Removing cachecontrol (0.12.11)
  • Removing certifi (2022.12.7)
  • Removing cffi (1.15.1)
  • Removing charset-normalizer (3.1.0)
  • Removing cleo (2.0.1)
  • Removing crashtest (0.4.1)
  • Removing distlib (0.3.6)
  • Removing dulwich (0.21.3)
  • Removing filelock (3.9.0)
  • Removing html5lib (1.1)
  • Removing idna (3.4)
  • Removing importlib-metadata (6.0.0)
  • Removing importlib-resources (5.12.0)
  • Removing installer (0.6.0)
  • Removing jaraco-classes (3.2.3)
  • Removing jsonschema (4.17.3)
  • Removing keyring (23.13.1)
  • Removing lockfile (0.12.2)
  • Removing more-itertools (9.1.0)
  • Removing msgpack (1.0.4)
  • Removing packaging (23.0)
  • Removing pexpect (4.8.0)
  • Removing pkginfo (1.9.6)
  • Removing pkgutil-resolve-name (1.3.10)
  • Removing platformdirs (2.6.2)
  • Removing poetry (1.4.0)
  • Removing poetry-core (1.5.1)
  • Removing poetry-plugin-export (1.3.0)
  • Removing ptyprocess (0.7.0)
  • Removing pycparser (2.21)
  • Removing pyproject-hooks (1.0.0)
  • Removing pyrsistent (0.19.3)
  • Removing rapidfuzz (2.13.7)
  • Removing requests (2.28.2)
  • Removing requests-toolbelt (0.10.1)
  • Removing shellingham (1.5.0.post1)
  • Removing six (1.16.0)
  • Removing tomli (2.0.1)
  • Removing tomlkit (0.11.6)
  • Removing trove-classifiers (2023.2.20)
  • Removing urllib3 (1.26.14)
  • Removing virtualenv (20.20.0)
  • Removing webencodings (0.5.1)
  • Removing xattr (0.10.1)
  • Removing zipp (3.15.0)

As you can see from this, poetry is installed into a completely different venv, and it's trying to remove itself. The value of virtualenvs.in-project does not matter (i'm just using it for easier cleanup). As mentioned above, it repros on several poetry releases, so it's obviously something somewhat fundamental. I know the system python on OSX is notoriously bad, but I was hoping could direct me more in debugging. Thanks!

[dimbleby]

not a lot of help I'm afraid, but I can report that your script does not have that effect for me (on ubuntu 22). The final poetry install --sync just does nothing, as you'd expect in an empty project with no dependencies.

probably not going to be a poetry bug I guess, but I don't think I can help you to debug it to be sure.

[novas0x2a]

I couldn't repro it on any non-osx environments I tried it on; I'm assuming it's something similar to the relocatable binary problems osx has had in the past, except I believe these tested poetry versions have the workarounds already

[novas0x2a]

I've successfully reproduced this with the system python 3.8.2 on osx 11.2.1 20D75 (big sur), so it's definitely not just the ancient 10.15 version

[novas0x2a]

The problem appears to be fixed on with python 3.9.6 on osx 13.2.1 22D68 (ventura).

[novas0x2a]

In the below, poetry is installed into the meta venv, and the project venv is project/.venv

Part 1

So, I added a print statement to poetry.utils.env.VirtualEnv:

    def __init__(self, path: Path, base: Path | None = None) -> None:
        super().__init__(path, base)

        # If base is None, it probably means this is
        # a virtualenv created from VIRTUAL_ENV.
        # In this case we need to get sys.base_prefix
        # from inside the virtualenv.
        if base is None:
            output = self.run_python_script(GET_BASE_PREFIX)
            assert isinstance(output, str)
            self._base = Path(output.strip())

        print(f"{path}: {self.sys_path}") # ADDED THIS

And what I see is:

working system

$ ~/meta/bin/poetry install --dry-run --sync
/home/mlundy/mlundy-test/test/project/.venv: ['/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/mlundy/mlundy-test/test/project/.venv/lib/python3.10/site-packages']
Installing dependencies from lock file

failing system

(note: as above, i've redacted the real HOME path with $HOME)

$ ~/mlundy-test/mlundy-test/meta/bin/poetry install --sync --dry-run
$HOME/mlundy-test/mlundy-test/test/project/.venv: ['/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip', '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8', '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload', '$HOME/mlundy-test/mlundy-test/meta/lib/python3.8/site-packages']
Installing dependencies from lock file

Package operations: 0 installs, 0 updates, 51 removals

  • Removing attrs (22.2.0)
  • Removing build (0.10.0)
etc

This shows that VirtualEnv.sys_path for the project venv is ending up with the meta venv's site-packages (which then, in the installer, results in it thinking poetry is installed in the project venv).

Part 2

This is particularly strange, because when I run the GET_SYS_PATH code directly using the venv python version, i see:

$HOME/mlundy-test/mlundy-test/test/project/.venv/bin/python -I -W ignore -c 'import json; import sys; print(json.dumps(sys.path))'
["/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip", "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8", "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload", "$HOME/mlundy-test/mlundy-test/test/project/.venv/lib/python3.8/site-packages"]

Whereas when I run it with the meta python:

$ ~/mlundy-test/mlundy-test/meta/bin/python -I -W ignore -c 'import json; import sys; print(json.dumps(sys.path))'
["/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip", "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8", "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload", "$HOME/mlundy-test/mlundy-test/meta/lib/python3.8/site-packages"]

In other words, maybe output = self.run_python_script(GET_SYS_PATH) appears to be using the wrong python?

Part 3

I edited VirtualEnv.run as follows:

    def run(self, bin: str, *args: str, **kwargs: Any) -> str | int:
        cmd = self.get_command_from_bin(bin) + list(args)
        print(f"{cmd}: {kwargs}")
        ret = self._run(cmd, **kwargs)
        print(f"    RETURN: {ret}")
        return ret

And reran:

working

['/home/mlundy/mlundy-test/test/project/.venv/bin/python', '-I', '-W', 'ignore', '-']: {'input_': 'import sys\n\nif hasattr(sys, "real_prefix"):\n    print(sys.real_prefix)\nelif hasattr(sys, "base_prefix"):\n    print(sys.base_prefix)\nelse:\n    print(sys.prefix)\n', 'stderr': -1}
    RETURN: /usr

['/home/mlundy/mlundy-test/test/project/.venv/bin/python', '-I', '-W', 'ignore', '-']: {'input_': 'import json\nimport sys\n\nprint(json.dumps(sys.path))\n', 'stderr': -1}
    RETURN: ["/usr/lib/python310.zip", "/usr/lib/python3.10", "/usr/lib/python3.10/lib-dynload", "/home/mlundy/mlundy-test/test/project/.venv/lib/python3.10/site-packages"]
...

failing

['$HOME/mlundy-test/mlundy-test/test/project/.venv/bin/python', '-I', '-W', 'ignore', '-']: {'input_': 'import sys\n\nif hasattr(sys, "real_prefix"):\n    print(sys.real_prefix)\nelif hasattr(sys, "base_prefix"):\n    print(sys.base_prefix)\nelse:\n    print(sys.prefix)\n', 'stderr': -1}
    RETURN: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8

['$HOME/mlundy-test/mlundy-test/test/project/.venv/bin/python', '-I', '-W', 'ignore', '-']: {'input_': 'import json\nimport sys\n\nprint(json.dumps(sys.path))\n', 'stderr': -1}
    RETURN: ["/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip", "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8", "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload", "$HOME/mlundy-test/mlundy-test/meta/lib/python3.8/site-packages"]

So, it's running with the correct python but somehow ending up with the wrong path, and running exactly the same command in part 2 and part 3 gets different answers.

Part 4

The environment, maybe?
I added

        for k, v in sorted(os.environ.items()):
            print(f"{k}={v}")

to VirtualEnv.run on the failing machine.

It's difficult to paste the outputs directly (lots of redaction required) but I compared the env debug outputs of the script with env | sort from the shell around the command in Part 2, and the only difference was:

$ diff -u env-before out 
--- sys-path-python-directly  2023-03-07 16:36:49.000000000 -0800
+++ sys-path-from-poetry 2023-03-07 16:38:51.000000000 -0800
@@ -3,6 +3,7 @@
 LOGNAME=<redacted>
 OLDPWD=$HOME/mlundy-test/mlundy-test/test
 PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
+PLAT=macosx-10.14.6-x86_64
 PS1=\h:\W \u\$ 
 PWD=$HOME/mlundy-test/mlundy-test/test/project
 SHELL=/bin/bash
@@ -13,4 +14,6 @@
 TERM=xterm-256color
 TMPDIR=<redacted>
 USER=<redacted>
-_=/usr/bin/env
+_=$HOME/mlundy-test/mlundy-test/meta/bin/poetry
+__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
+__PYVENV_LAUNCHER__=$HOME/mlundy-test/mlundy-test/meta/bin/python3

Oh, a smoking gun. That environment variable led me to tox-dev/tox#148
I added os.environ.pop("__PYVENV_LAUNCHER__", None) to VirtualEnv.run and... it worked!

[novas0x2a]

So, I'm not sure what to do with this information because I'm not quite sure what this env var is for, other than everyone seems to remove it:

• Fix provisioning from pyvenv interpreter tox-dev/tox#1452
• https://virtualenv.pypa.io/en/20.0.33/changelog.html#features-20-0-14
• bpo-22490: Remove __PYVENV_LAUNCHER__ from environment during launch python/cpython#9516

Obviously, this problem is fixed in later python versions, but, would you consider putting a workaround for this into poetry as well, since the affected OSX versions are still in the wild?

[novas0x2a]

OH. Okay. I found the bug. --^

[TiansuYu]

The issue still exists for a conda managed virtual environment with python3.10 and poetry1.5.1.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.