Once system python and pipx upgraded, cannot install new app if some previous old shared_venv existed

[hcoohb]

Describe the bug

On Manjaro/Arch, I was with python 3.7.4.
To ensure stability of some apps (and as some apps are not fully working on 3.8 that I new was coming) I also had python 3.7.5 installed through pyenv.
I use this python to install my apps with

pipx install --python ~/.pyenv/versions/3.7.5/bin/python flexget
pipx inject flexget transmissionrpc

Then my system got upgraded from 3.7.4 to 3.8 (Note that my 3.7.5 pyenv stayed)
I reinstalled pipx on 3.8 on system.
Then I had error thrown when installing new pipx no matter what versions I tried:

$ pipx install pycowsay --verbose
pipx > (run_pipx_command:134): Virtual Environment location is /home/hcooh/.local/pipx/venvs/pycowsay
pipx > (run:97): running /usr/bin/python -m venv --without-pip /home/hcooh/.local/pipx/venvs/pycowsay
pipx > (run:97): running /home/hcooh/.local/pipx/venvs/pycowsay/bin/python -m pip install pycowsay
/home/hcooh/.local/pipx/venvs/pycowsay/bin/python: No module named pip

pipx > (rmdir:16): removing directory /home/hcooh/.local/pipx/venvs/pycowsay
'/home/hcooh/.local/pipx/venvs/pycowsay/bin/python -m pip install pycowsay' failed

or even trying to use the pyenv:

pipx install --python ~/.pyenv/versions/3.7.5/bin/python pycowsay --verbose
pipx > (run_pipx_command:134): Virtual Environment location is /home/hcooh/.local/pipx/venvs/pycowsay
pipx > (run:97): running /home/hcooh/.pyenv/versions/3.7.5/bin/python -m venv --without-pip /home/hcooh/.local/pipx/venvs/pycowsay
pipx > (run:97): running /home/hcooh/.local/pipx/venvs/pycowsay/bin/python -m pip install pycowsay
/home/hcooh/.local/pipx/venvs/pycowsay/bin/python: No module named pip

pipx > (rmdir:16): removing directory /home/hcooh/.local/pipx/venvs/pycowsay
'/home/hcooh/.local/pipx/venvs/pycowsay/bin/python -m pip install pycowsay' failed

Running the already installed app (using the 3.7.5) would work fine and even pipx runpip xxx list
I realised then that .local/pipx/shared/pyvenv.cfg mentioned python version 3.7.4.
So I renamed the .local/pipx/shared folder differently. It recreated the shared lib for python3.8.
I had to then symlink the pyenv folder of python3.7 with:

ln -s ../../../../.pyenv/versions/3.7.5/lib/python3.7/ ~/.local/pipx/shared/lib/python3.7

And then install run fine and the already installed app run fine too

How to reproduce

See above

Expected behavior

pipx Install install should work after a pyhton upgrade even if hte previous shared folder still exists.
i think

[itsayellow]

Hm. I'm trying to figure out how I might reproduce this on my own system. There's a lot going on in your issue description.

Do you have a simplified way to reproduce this that I could try, possibly using pyenv?

For the devs, it appears that Venv.create_venv() is being called, which would execute shared_libs.create(). This happens without error, so either its creating a shared library dir or finding one it thinks is valid.

Since it can't find pip, it seems as though either pipx_shared.pth is absent, in the wrong place, or contains a bad path to the shared libraries. Or something out of my conception...?

Unless I can somehow reproduce this on my system I'm pretty lost as to what is going on.

[darkrain42]

@itsayellow, I'm not sure this is reproducible with pyenv, but I can detail how I encountered it / can reproduce it on Debian (where sid recently went from /usr/bin/python3 being 3.7 to 3.8) without anything other than the system Python, and maybe that will help.

In detail:

• docker run --rm -it debian:bullseye # this will stop working when python3.8 migrates to Debian testing, but it's faster than upgrading from buster.

(All steps now inside the container)

• apt -y update && apt -y install --no-install-recommends python3-pip python3-setuptools python3-venv python3-wheel
• python3 -m pip install pipx
• pipx install yq

yq is now in ~root/.local/pipx/venv

Update python3 to python3.8:

• sed -i -e 's/bullseye/sid/g' /etc/apt/sources.list
• apt -y update && apt -y upgrade

root@e861fc867d19:/# python3 --version
Python 3.8.2
root@e861fc867d19:/# pipx
Traceback (most recent call last):
  File "/usr/local/bin/pipx", line 6, in <module>
    from pipx.main import cli
ModuleNotFoundError: No module named 'pipx'
root@e861fc867d19:/# 

Reinstall pipx:

• python3 -m pip install pipx

At this point, trying to upgrade an existing app or install a new one fail with a "no module named pip" error

root@e861fc867d19:/# pipx upgrade yq
/root/.local/pipx/shared/bin/python: No module named pip
Failed to upgrade shared libraries
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pipx/shared_libs.py", line 70, in upgrade
    run(
  File "/usr/local/lib/python3.8/dist-packages/pipx/util.py", line 134, in run
    raise PipxError(f"{cmd_str!r} failed")
pipx.util.PipxError: '/root/.local/pipx/shared/bin/python -m pip --disable-pip-version-check install -q --upgrade pip setuptools wheel' failed
/root/.local/pipx/venvs/yq/bin/python: No module named pip
'/root/.local/pipx/venvs/yq/bin/python -m pip install --upgrade yq -q' failed
root@e861fc867d19:/# 

# pipx install --verbose jq
pipx > (_package_name_from_spec:93): Determined package name: jq
pipx > (_package_name_from_spec:94): Package name determined in 0.0s
pipx > (run_subprocess:112): running /usr/bin/python3 -m venv --without-pip /root/.local/pipx/venvs/jq
pipx > (run_subprocess:112): running /root/.local/pipx/venvs/jq/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /root/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /root/.local/pipx/venvs/jq/bin/python --version
pipx > (run_subprocess:112): running /root/.local/pipx/venvs/jq/bin/python -m pip install jq
/root/.local/pipx/venvs/jq/bin/python: No module named pip
pipx > (install_package:189): '/root/.local/pipx/venvs/jq/bin/python -m pip install jq' failed

pipx > (rmdir:18): removing directory /root/.local/pipx/venvs/jq
Error installing jq.
root@e861fc867d19:/# 

[itsayellow]

Have you tried pipx reinstall-all? This command is supposed to help for situations when the system python changes, and has been known to work for that situation on the Mac.

If that doesn't work, what does pipx list return? It should show a lit of packages with warnings that they are missing a valid python interpreter.

Did you uninstall the old jq before trying to install the new jq? That may be important.

The other thing I can think of that might possibly be affecting you is that you are installing pipx to the system as opposed to using python3 -m pip install --user pipx as is our recommendation.

[darkrain42]

It's subtle (I should probably have picked better example packages for the demo), but note that the first package was yq (YAML) and the second was jq (JSON). Sorry :)

• pipx reinstall-all fails during the 'install' portion of reinstalling the first package. (Leaving the package uninstalled, and losing pipx_metadata.json!)
• pipx list works fine
• If I run the pip / pipx portions of the above as a non-root user, it still reproduces.

Before upgrading the system Python:

root@cc012cba1af1:/# su - paul
paul@cc012cba1af1:~$ python3 -m pip install pipx
...
paul@cc012cba1af1:~$ export PATH="$PATH:$HOME/.local/bin"
paul@cc012cba1af1:~$ which pipx
/home/paul/.local/bin/pipx
paul@cc012cba1af1:~$ pipx install yq
  installed package yq 2.10.0, Python 3.7.6
  These apps are now globally available
    - xq
    - yq
done! ✨ 🌟 ✨
paul@cc012cba1af1:~$

After upgrading the system Python:

root@cc012cba1af1:/# su - paul
paul@cc012cba1af1:~$ export PATH="$PATH:$HOME/.local/bin"
paul@cc012cba1af1:~$ pipx
Traceback (most recent call last):
  File "/home/paul/.local/bin/pipx", line 6, in <module>
    from pipx.main import cli
ModuleNotFoundError: No module named 'pipx'
paul@cc012cba1af1:~$ python3 -m pip install pipx
Collecting pipx
...
Successfully installed argcomplete-1.11.1 click-7.1.1 distro-1.4.0 pipx-0.15.1.3 userpath-1.3.0
paul@cc012cba1af1:~$ pipx list
venvs are in /home/paul/.local/pipx/venvs
apps are exposed on your $PATH at /home/paul/.local/bin
   package yq 2.10.0, Python 3.7.6
    - xq
    - yq
paul@cc012cba1af1:~$ pipx install jq
/home/paul/.local/pipx/venvs/jq/bin/python: No module named pip

Error installing jq.
paul@cc012cba1af1:~$ pipx reinstall-all --verbose
pipx > (needs_upgrade:51): Time since last upgrade of shared libs, in seconds: 144.02421402931213. Upgrade will be run by pipx if greater than 2592000.0.
pipx > (needs_upgrade:51): Time since last upgrade of shared libs, in seconds: 144.0251932144165. Upgrade will be run by pipx if greater than 2592000.0.
pipx > (uninstall:292): removing symlink /home/paul/.local/bin/yq
pipx > (uninstall:292): removing symlink /home/paul/.local/bin/xq
pipx > (rmdir:18): removing directory /home/paul/.local/pipx/venvs/yq
uninstalled yq! ✨ 🌟 ✨
pipx > (run_subprocess:112): running /usr/bin/python3 -m venv --without-pip /home/paul/.local/pipx/venvs/yq
pipx > (run_subprocess:112): running /home/paul/.local/pipx/venvs/yq/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/paul/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/paul/.local/pipx/venvs/yq/bin/python --version
pipx > (run_subprocess:112): running /home/paul/.local/pipx/venvs/yq/bin/python -m pip install yq
/home/paul/.local/pipx/venvs/yq/bin/python: No module named pip
pipx > (install_package:189): '/home/paul/.local/pipx/venvs/yq/bin/python -m pip install yq' failed

pipx > (rmdir:18): removing directory /home/paul/.local/pipx/venvs/yq
Error installing yq.
paul@cc012cba1af1:~$ pipx list
nothing has been installed with pipx 😴
paul@cc012cba1af1:~$

Some further bits of information:

This seems to be specific to using a python3 -m pip (or pip3) to install pipx, which causes pipx's shebang to be #!/usr/bin/python3.

When I use python3.7 -m pip install pipx, then pipx's shebang is versioned (#!/usr/bin/python3.7), and as long as Debian's python3.7 package is still installed, everything keeps working.

Further, if I then uninstall python3.7 (apt -y autoremove --purge) and reinstall pipx (python3.8 -m pip install pipx), pipx is able to detect it needs to reinitialize the shared packages (note the python3.8 -m venv --clear /home/paul/.local/pipx/shared below)

root@624077171e9c:/# su - paul
paul@624077171e9c:~$ pipx --verbose
-bash: /home/paul/.local/bin/pipx: /usr/bin/python3.7: bad interpreter: No such file or directory
paul@624077171e9c:~$ python3.8 -m pip install pipx
Collecting pipx
...
Successfully installed argcomplete-1.11.1 click-7.1.1 distro-1.4.0 pipx-0.15.1.3 userpath-1.3.0
paul@624077171e9c:~$ pipx install --verbose jq
pipx > (_package_name_from_spec:93): Determined package name: jq
pipx > (_package_name_from_spec:94): Package name determined in 0.0s
pipx > (run_subprocess:112): running /usr/bin/python3.8 -m venv --without-pip /home/paul/.local/pipx/venvs/jq
pipx > (run_subprocess:112): running /usr/bin/python3.8 -m venv --clear /home/paul/.local/pipx/shared
pipx > (upgrade:62): Upgrading shared libraries in /home/paul/.local/pipx/shared
pipx > (run_subprocess:112): running /home/paul/.local/pipx/shared/bin/python -m pip --disable-pip-version-check install --upgrade pip setuptools wheel
Collecting pip
...
Successfully installed pip-20.0.2 setuptools-46.0.0 wheel-0.34.2
pipx > (run_subprocess:112): running /home/paul/.local/pipx/venvs/jq/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/paul/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/paul/.local/pipx/venvs/jq/bin/python --version
pipx > (run_subprocess:112): running /home/paul/.local/pipx/venvs/jq/bin/python -m pip install jq
Collecting jq
  Downloading jq-0.1.6.tar.gz (35 kB)
...

[itsayellow]

OK, I finally had a chance to run your very complete docker instructions. Thank you for that!

I can reproduce what you're talking about. And it doesn't seem to help if you install pipx using pip install --user.

First problem, yq stops working after system python upgrade

What seems to be happening is that in ~/.local/pipx/venv/yq/bin all the pythons are being linked back only to /usr/bin/python3. This is not what happens for me on my Mac: there each venv's python is linked to a very particular version of python on my system (e.g. /usr/local/Cellar/python/3.7.7/python3.7.). So it doesn't come up empty looking for a non-existent python (which we handle), but the python it finds has a different minor version than before. This causes site packages to be in the wrong place (they are in ~/.local/pipx/venvs/yq/lib/python3.7/ but the new python3.8 is looking for ~/.local/pipx/venvs/yq/lib/python3.8/ which doesn't exist)

I see the same behavior on my Ubuntu system. The python interpreter link in all the venvs points to /usr/bin/python3, not a specific version like /usr/bin/python3.7.

The workaround when you first install yq would be this pipx command:

pipx install --python /usr/bin/python3.7 yq

Then after upgrading the system python yq will still work. (I tried it and it works for me.)

My suggestion to the pipx developers is that when we determine the system python, maybe we should resolve the link back to the original binary filepath which would get a specific python sub-version. (e.g. in utils.py we should use DEFAULT_PYTHON = Path(sys.executable).resolve()). I'm not sure if that's the best solution (it would cause sub-minor python upgrades to break pipx venvs) but it's a solution.

Second problem, reinstall of pipx will not install after system upgrade

I think this is similar to the venv problem described above. The shared libs have their own idea of a python binary, and on linux this seems to be linked to a generic binary without a sub-version (i.e. /usr/bin/python3) which then screws up the shared lib venv when the python minor version number changes. It appears to pipx that at least the python binary is valid, but the shared_lib venv has no site-packages under the the new python3.8 only under the old python3.7.

shared_libs.py, _SharedLibs.is_valid() only checks if we have files of the python binary and the pip binary, but it doesn't check that the library pip is still accessible for the current python. When the python the shared_libs is linked to gets changed from 3.7 to 3.8, then we no longer have the package pip under the right directory in the shared_libs venv.

[itsayellow]

btw the workaround after an upgrade of the system python to get pipx working (after reinstalling pipx):

rm ~/.local/pipx/shared/bin/python3
ln -s /usr/bin/python3.7 ~/.local/pipx/shared/bin/python3

Or at least I assume that should be the workaround, but when I tried it, it found pip but then failed spectacularly trying to build a wheel for jq.

[itsayellow]

It seems that we need a better _SharedLibs.is_valid() that is more in-depth and can trigger a reinstall of the shared libs if there is no pip library found.

[darkrain42]

Or at least I assume that should be the workaround, but when I tried it, it found pip but then failed spectacularly trying to build a wheel for jq.

Sorry, that's another example of "I should have chosen better packages as examples." It fails because of no C compiler.

Using "yq" and then "httpie" seem like better example packages -- neither require compilation.

[metinsenturk]

I think I encountered a similar problem after upgrading my python install from the brew. The solution that worked for me is pipx reinstall-all.

To illustrate the problem and how it gets resolved, the following are the commands I did.

 me@MacBook-Pro  ~  python --version
python 3.8.3

 me@MacBook-Pro  ~ brew upgrade
...
all done!
 me@MacBook-Pro  ~ python --version 
python 3.8.5
 me@MacBook-Pro  ~  pipx list
venvs are in /Users/me/.local/pipx/venvs
apps are exposed on your $PATH at /Users/me/.local/bin
   package autopep8 has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8
   package black has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8
   package flake8 has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8
   package jupyterlab has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8
   package pycowsay has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8
   package pylint has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8
   package virtualenv has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8

 me@MacBook-Pro  ~  pipx reinstall-all
...
 me@MacBook-Pro  ~  pipx list
venvs are in /Users/me/.local/pipx/venvs
apps are exposed on your $PATH at /Users/me/.local/bin
   package autopep8 1.5.4, Python 3.8.5
    - autopep8
   package black 20.8b1, Python 3.8.5
    - black
    - black-primer
    - blackd
   package flake8 3.8.4, Python 3.8.5
    - flake8
   package jupyterlab 2.2.8, Python 3.8.5
    - ...
   package pycowsay 0.0.0.1, Python 3.8.5
    - pycowsay
   package pylint 2.6.0, Python 3.8.5
    - ...
   package virtualenv 20.0.33, Python 3.8.5
    - virtualenv

[itsayellow]

This last example is related to #493

[sean-abbott]

This may or may not useful, but this also happens if you do a fresh install on a new computer and a higher version of ubuntu (or probably any other linux), and then transfer your home directory.

pipx get very mad, and the only way I was able to fix it was to wipe everything pipx related and start reinstalling, which is less than ideal.

I worked my way through several of the folks examples here, but in the end I think I made it worse. Only by removing everything under ~/.local/pipx was I able to start installing from scratch again.

[itsayellow]

@sean-abbott depending on how old your original pipx was, this may also be because pipx has been extensively upgraded in the past year or so. Usually pipx reinstall-all can fix most issues.

[sean-abbott]

I tried running reinstall-all, it gave me errors. Unfortunately I have both closed the terminal and "fixed" it by just reinstalling, so I can't give you the errors anymore, I'm sorry.

…

On Sat, Dec 26, 2020 at 8:47 PM Matthew Clapp ***@***.***> wrote: @sean-abbott <https://github.com/sean-abbott> depending on how old your original pipx was, this may also be because pipx has been extensively upgraded in the past year or so. *Usually* pipx reinstall-all can fix most issues. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#294 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKWJVYILJ5MK3PZW3RUR53SW2G2DANCNFSM4J26VQBA> .

[michjnich]

I'm not sure if this is related, but I've had very similar issues. After much un/re-installing with manual clear downs of the shared dirs I've got rid od f the "can't find pip" issue, but am now having the following when trying to install something:

$  pipx install flake8
Error: Command '['/home/mike/.local/pipx/shared/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

'/usr/bin/python -m venv --clear /home/mike/.local/pipx/shared' failed

I can uninstall pipx, remove the .local/pipx tree completely, then reinstall and try it, and it's the same problems.

This following an upgrade from 3.8 => 3.9 in Ubuntu on WSL2, using update-alternatives to manage the python version.

If I flip the python version back to 3.8 via update-alternatives, install pipx for that, then it all works just fine and I can pipx install as expected.

Then going back to 3.9 requires installing pipx again. After that pipx listworks, but nothing else - then I get "No module named pip".

[itsayellow]

@michjnich , what happens with Python 3.9 if you just try the following on your command-line:

pip --version

If that's missing, then you might need to use apt to reinstall for Python 3.9 the necessary modules for Ubuntu that ensure venv and pip are available:
https://pipxproject.github.io/pipx/troubleshooting/#debian-ubuntu-issues

sudo apt install python3-venv python3-pip

[michjnich]

Seems like it's pointing at the correct version already.:

$   python --version
Python 3.9.1
$   pip --version
pip 21.0 from /home/mike/.local/lib/python3.9/site-packages/pip (python 3.9)

[itsayellow]

OK, the other important package to check is python3-venv

> apt list --installed python3-venv    
Listing... Done
python3-venv/focal,now 3.8.2-0ubuntu2 amd64 [installed]
> apt list --installed python3-pip     
Listing... Done
python3-pip/focal-updates,focal-updates,focal-security,focal-security,now 20.0.2-5ubuntu1.1 all [installed]
N: There is 1 additional version. Please use the '-a' switch to see it

Hopefully with Python 3.9 installed you should get each package listed as you query the apt list command and [installed] next to them.

[michjnich]

$  python --version
Python 3.9.1
$  apt list --installed python3-venv
Listing... Done
python3-venv/focal,now 3.8.2-0ubuntu2 amd64 [installed]
$  apt list --installed python3-pip
Listing... Done
python3-pip/focal-updates,focal-security,now 20.0.2-5ubuntu1.1 all [installed]
N: There is 1 additional version. Please use the '-a' switch to see it

So they're installed, but should venv be showing a 3.9 version here?

[darkrain42]

So they're installed, but should venv be showing a 3.9 version here?

How did you install Python 3.9? You probably want python3.9-venv installed.

python3-venv is a dependency package which points at (on focal) python3.8-venv, so you may not automatically pick up the venv package without installing it manually.

[michjnich]

It was installed from the deadsnakes:ppa, then configured as an alternative version using update-alternatives (which is how I switch between system versions if I need to). I guess the 3.9-venv needs installing specifically here ... I'll get that set up and see if it resolves the issue. Thanks!

[NotAFile]

Just had this issue again upgrading to fedora 35:

/home/nota/.local/pipx/shared/bin/python: No module named pip
Failed to upgrade shared libraries
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pipx/shared_libs.py", line 113, in upgrade
    subprocess_post_check(upgrade_process)
  File "/usr/lib/python3.10/site-packages/pipx/util.py", line 203, in subprocess_post_check
    raise PipxError(
pipx.util.PipxError: '/home/nota/.local/pipx/shared/bin/python -m pip --disable-pip-version-check install -q --upgrade pip setuptools wheel' failed
/home/nota/.local/pipx/venvs/pyupgrade/bin/python: No module named pip
Error encountered when upgrading pyupgrade:
'/home/nota/.local/pipx/venvs/pyupgrade/bin/python -m pip install --upgrade pyupgrade -q' failed

I fixed it by running:

rm -r ~/.local/pipx/shared/

and then running a pipx operation to make it recreate the shared env with the new python version.

[nyanpasu64]

Arch Linux, pipx 0.16.4, just upgraded from Python 3.9 to 3.10, and as per tradition all my pipx packages broke.

Old venvs break

running a pipx operation to make it recreate the shared env with the new python version.

Even after recreating the shared env, some of my existing envs don't load, since they only have ~/.local/pipx/venvs/PACKAGE/lib/python3.9/:

> tg
Traceback (most recent call last):
  File "/home/nyanpasu64/.local/bin/tg", line 5, in <module>
    from tg.__main__ import main
ModuleNotFoundError: No module named 'tg'

> aws
Traceback (most recent call last):
  File "/home/nyanpasu64/.local/bin/aws", line 19, in <module>
    import awscli.clidriver
ModuleNotFoundError: No module named 'awscli'

> scan-build
Traceback (most recent call last):
  File "/home/nyanpasu64/.local/bin/scan-build", line 5, in <module>
    from libscanbuild.analyze import scan_build
ModuleNotFoundError: No module named 'libscanbuild'

Interestingly, maestral and corrscope (which I installed with --system-site-packages so they could use system PyQt5) work without having to reinstall. It seems pipx upgrade-all somehow produced a corrscope/lib/python3.10 with very recent modification dates (11:02:06 PM), and maestral/lib/python3.10 (11:02:00 PM). For context, I updated python at 10:57 PM, tried and failed to run pipx upgrade-all at 11:01:57 PM, and recreated my pipx shared env later at 11:02:39 PM. Interestingly, corrscope/bin and maestral/bin still have python3.9 binaries, but they now point to /usr/bin/python3.10 (possibly because /usr/bin/python links to /usr/bin/python3.10).

However, corrscope (which I installed with now fails to run when I uninstall system matplotlib (prints ModuleNotFoundError: No module named 'matplotlib'), because the pipx-venv matplotlib contains CPython 3.9 extensions and can no longer be loaded by 3.10.

Reinstalling works-ish

pipx reinstall awscli worked. I tried reinstalling this using fd -d 1 . ~/.local/pipx/venvs/ --exec pipx reinstall '{/}' (because I didn't realize that reinstall-all existed). The uninstalls succeeded, but the installs failed:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/nyanpasu64/.local/pipx/shared/lib/python3.10/site-packages/pip/__main__.py", line 9, in <module>
    if sys.path[0] in ("", os.getcwd()):
FileNotFoundError: [Errno 2] No such file or directory

I think that fd doesn't change the current working directory, implying that pipx reinstall failed because I ran fd in a folder deleted by one of the reinstall commands. Do you consider it a bug that pipx fails when run in a folder deleted by pip install, and do you plan to cd $HOME after deleting the venv? Is it a bug that a failed pipx reinstall deletes the venv folder and makes pipx act like the package never existed?

Suggestions

• Change pipx reinstall-all to first delete (or rename) and recreate the shared venv, before reinstalling all packages from the original spec. (Before I realized that reinstall-all existed, I wrote: ideally if reinstalling any package fails, the old environment is left in place, and/or the failing packages are printed.) IDK if this is implemented or necessary.
• Make pipx reinstall not fail if your current working directory is deleted by the uninstall process.
• Somehow make pipx reinstall better handle the case where reinstallation fails (perhaps leave the old folder in place? perhaps rename the old folder instead of deleting it?).

[Exagone313]

I reinstalled pipx using pip install pipx after upgrading to Python 3.10.

When I ran pipx reinstall-all, it printed a lot of errors and removed all my virtualenvs, it failed to reinstall these.

https://gist.github.com/Exagone313/99b7aa35b8aec098f431e702d9416f35

After that, running the command again gave an error about the pip package being not found in pipx virtualenv:

/home/emartinet/.local/pipx/shared/bin/python: No module named pip
Failed to upgrade shared libraries
Traceback (most recent call last):
  File "/home/emartinet/.local/lib/python3.10/site-packages/pipx/shared_libs.py", line 113, in upgrade
    subprocess_post_check(upgrade_process)
  File "/home/emartinet/.local/lib/python3.10/site-packages/pipx/util.py", line 165, in subprocess_post_check
    raise PipxError(
pipx.util.PipxError: '/home/emartinet/.local/pipx/shared/bin/python -m pip --disable-pip-version-check install -q --upgrade pip setuptools wheel' failed

I solved it by moving the ~/.local/pipx directory: mv ~/.local/pipx{,.old}, then I reinstalled all my packages manually.

---

I think the reinstallation should be protective and keep the list of installed virtualenvs at some place to deal with installation errors. At least I had the log.

[nyanpasu64]

Summary

Judging from #294 (comment) and #294 (comment), there's at least 3 versions in a pipx setup:

• The Python interpreter version used to install pipx
• The shared env's version
• Each app's venv version

Each of these should match the system Python interpreter, but might desync (the system Python interpreter's path is still present, but belongs to a different Python version) when updating Python in-place on Linux.

Ideas

Not a pipx contributor so far, but I could try changing reinstall-all to always recreate the shared environment, and change more pipx operations (reinstall, upgrade[-all], install, etc.) to automatically recreate the shared environment if the Python interpreter version has changed. (Perhaps it should also reinstall all packages? It's necessary to fix broken packages, but unexpected by the user. Maybe do nothing in shell scripts and prompt the user if a TTY is detected?) You'll either have to save the install-time Python version in a config file, or detect the install-time Python version from the shared/app venv (I'm not sure how, and looking for a .../lib/python3.x folder can mistakenly indicate a venv was installed on newer Python than it actually was; on current pipx I managed to end up with a shared env with an empty python3.10 folder, and an app's venv with a python3.10 folder containing Python 3.10 __pycache__ but 3.9 C extensions).

My other idea is to add a config file/database containing a list of venvs and each venv's install options and Python versions. This complicates the architecture since the config file can desync with the actual folders present (eg. if a user adds/removes venv folders by hand), but is useful since if a failed reinstall deletes a venv, it's still present in the config/database and a subsequent reinstall-all knows to install them. This config file should also store the shared environment's Python version and/or interpreter path, so pipx can compute when to regenerate the shared env and all app venvs (or you can skip this and use the version stored in ~/.local/pipx/shared/pyvenv.cfg).

In theory you could also store the Python version/path of each app's venv (though this is already tracked in ~/.local/pipx/venvs/*/pyvenv.cfg. Using this to support different venvs with different Python versions/interpreters is too complex IMO. It might be useful to use this to track which venvs need to be reinstalled. I kinda prefer not supporting partially upgrading venvs, but instead block all pipx operations until the shared env and all venvs are upgraded in a single operation. But old pipx versions (combined with users copying commands to try to fix pipx) can generate mixed Python versions. If we don't check per-venv Python versions, we can update the docs and tell users to "reinstall-all to fix things"; if we do, we can detect and address this situation.

I think it's still a good idea for reinstall-all to regenerate the shared env, even if it's supposedly up-to-date according to the "install-time Python version" metadata. This allows the command to fix errors tied to the shared env, rather than any app's venv. (These errors may exist, I've never encountered any other than a Python version mismatch.) This also fixes cases where the Python version has actually changed even though the metadata wrongly indicates the install-time version matches the current version.

Is it possible for each app's shim binary to automatically reinstall the app (and shared env, maybe other apps) if necessary? This makes the reinstallation process transparent to users and hopefully "just work", but I'm not sure if users want their apps to magically change behavior in edge cases (though they already break right now). And this might slow-down happy-path app startup time (though you can get pretty close to zero overhead by only reinstalling when you catch ModuleNotFoundError) (though I don't know if this fixes both "outdated shared env and app" and "updated shared env but outdated app"). And this doesn't catch cases where pipx install --system-site-packages and pipx upgrade results in a working app which uses C libraries from the system rather than the venv.

Or perhaps pipx itself should catch ModuleNotFoundError: No module named 'pipx' and reinstall the shared env and all packages. (I don't think I've seen this particular error since I installed pipx from the Arch repositories rather than pip install --user.) This fixes another user-facing failure mode of pipx, but can't replace my previous suggestions since it fails to catch cases where pipx is updated but the shared env is out-of-date and unusable.

I didn't look into handling the system Python interpreter being uninstalled, and a new one being installed to a different location. This case is more common on Windows where Python is installed to C:\Python3xx\bin\python rather than /usr/bin/python. This may break the pipx binary, and require the user to reinstall pipx by hand, and then ideally pipx updates the shared and app envs from there.

Testing

I think it will be difficult to come to a consensus on an optimal design (probably manual testing will help in designing an intuitive interface), and difficult to manually or automatically test updating Python to a new major version (which only happens once per year per Linux installation). Perhaps pyenv will help to test switching Python interpreters quickly. There's also "build a Docker image with an outdated Python", but I'm less familiar with that than pyenv, and setting up Docker/Podman requires more upfront setup and invasive system changes (cgroups etc.) than pyenv IMO.

[gaborbernat]

We don't plan to support this for now. On python upgrades I recommend reinstall-all.