All virtualenvs break when Python version is upgraded #146
Comments
|
I wonder if pipx should use the |
|
Personally, I like symlinks and would prefer to keep them. I wonder instead of |
|
Another fix for the mean time would be to make a symlink where the old version of python was that points to the new one, then reinstall all. The |
|
I see |
|
Looks like |
|
I've run into this a few times. My solution has been to manage a One thing I've noticed (and even looked through the code a bit to try to resolve) is that either pipx or venv itself will dereference the path to the realpath. If I use my I've been meaning to look into exactly how pipx handles this realpath resolution (or if venv does this itself), since it would be nice to have a more stable pipx that doesn't break on random python upgrades. Admittedly, this is rare with pyenv (unlike homebrew now), but I've had it happen before. |
Doesn't work, unfortunately: I assume that's because the upgrade was not "in-place" (which is never going to be the case with Homebrew). |
|
I ran into this also, and used something like this to repair broken venvs. I don't think it's necessarily the right or best way to handle the problem, or at the very least it would need to be fleshed out a bit (there's no error handling or usage info, sloppily hardcoded to one python version, etc). But it's something that works in a pinch and could be the bones of something smarter. VENV_DIR="${1:-${HOME}/.local/pipx/venvs}"
# Delete any symlinks that point nowhere
find -L "$VENV_DIR" -type l | xargs rm
# Assume that a directory is a broken virtual environment if
# after the symlink cleanup:
#
# 1. bin/activate exists
# 2. bin/python does not exist
#
# And for those cases, create a fresh virtual environment.
for dir in $(find "$VENV_DIR" -type d -depth 1); do
if [[ -f "$dir/bin/activate" && -d "$dir/lib/python3.7" && ! -f "$dir/bin/python" ]]; then
echo "Refreshing Python virtual environment for $dir..."
python3 -m venv $dir
fi
doneI think this is fundamentally similar to OP's workaround, just operates one level lower (as a venv repair rather than a pipx reinstall). |
|
assuming this is specific to homebrew, an alternative solution is to suggest users set details: https://discourse.brew.sh/t/when-did-brew-upgrade-and-reinstall-start-removing-versions/4054 FWIW, all virtualenvs that directly depend on a particular version of a homebrew python will break if that version of python is deleted so this problem isn't specific to pipx |
|
I also hit this problem. All of the solutions listed here feel more like workarounds to me. Users shouldn't have to think about this. Anyone understands why the symlink to python3.X is automatically getting expanded, and which bit of code is doing that? |
|
I've just uninstalled the homebrew version of pipx due to this error. I don't know if anyone else noticed the connection but by default It actually surprised me that If this is going to work reliably with homebrew, it has to use a stable python target, like the system python, a python installed using the python.com installer, or one provided by pyenv. You cant swap virtualenv symlinks around reliably, ( it works sometimes, but don't let that fool you 😉 ) and as @chrish42 pointed out, things like Edit: Ive just done more digging and it looks like this came up before... From the comments on those issues, it looks like this issue might require some careful choices about just which python is chosen, and perhaps more explicit warnings about how an environment has to be setup, such as |
|
I just ran into this issue. I was probably using
But, this lead to the unpleasant situation of all my I think that the choice to symlink the interpreter is a misfeature, partly because of its fragility, and partly because it is different from the default behavior of That said: if the team wants |
|
I just ran into this as well, looks like my homebrew python got upgraded from 3.7.3 to 3.7.4 and broke things: Following for a solution.. Too bad though, pipx has been super easy to work with up until this point. |
|
I've just though of a particularly aggressive solution to the problem. It requires doing two things.
Thoughts? |
|
Using PyOxidizer sounds like it might work, and that would cover the second bullet point automatically. Things it might make harder:
TBH I don't have the time to look into this, but am not opposed to it if someone makes it happen. |
|
@cs01 Building on each OS shouldn't be a huge issue, your are right about there being free CI tools to handle this. As for the question of At the moment, I think the biggest complication/question is where/how we get the python versions. Borrowing the way that |
|
@techdragon pipx recently changed to use a single shared copy of pip and setuptools (look for |
|
It seems that To me this seems like an acceptable recovery of pipx's packages after a system python upgrade. Or at least much better than now. |
|
@pfmoore Thanks for filling me in there 😄 @itsayellow Your idea sounds less disruptive than mine would be, so its probably worth exploring that approach first to see if it can fix things. |
Fixes pypa#146 by allowing reinstall-all() to call uninstall() without breaking.
Fixes pypa#146 by allowing reinstall-all() to call uninstall() without breaking.
This fixes a fatal error that would come with a invalid python symlink in an existing shared library venv. This is necessary for reinstall-all to be used successfully to upgrade a system python (Issue pypa#146) Since we are creating the shared libraries over again, it is also just cleaner to allow things to start from scratch.
|
I use the following and this helps, I can gist this later into a shell script basically reinstall |
This fixes a fatal error that would come with a invalid python symlink in an existing shared library venv. This is necessary for reinstall-all to be used successfully to upgrade a system python (Issue #146) Since we are creating the shared libraries over again, it is also just cleaner to allow things to start from scratch.
|
Thanks @Linuxbytes. I personally haven't run into this problem since I don't use pyenv, so I haven't tried your solution. Anyone else? Another thing to look into is https://pypi.org/project/pipipxx/. It looks like it might help work around this issue. (see the text highlighted in bold below)
I'd be interested to hear if installing pipx with pipipxx makes upgrading python versions easier. |
Fixes pypa#146 by allowing reinstall-all() to call uninstall() without breaking.
Fixes pypa#146 by allowing reinstall-all() to call uninstall() without breaking.
|
I seem to be having this problem after upgrading python from 3.7 to 3.8 on macos via homebrew 2.2.2. I don't have a lot of packages installed with pipx so reinstalling them (via |
|
The current best way to fix a homebrew upgrade of the system python would be: If you installed or reinstalled your packages after pipx version 0.15.0.0, then all options, package specs, and injected packages will be remembered and |
|
Even when homebrew upgrade's python, it can retain the old version as long as homebrew doesn't clean up. I think it may be reasonable to request homebrew maintainers in the future to never cleanup certain formulae, such as a python. In the mean time, this environment variable prevents homebrew cleanup: With this enabled, when upgrading to 3.7.6, the 3.7.5 is still available at this path: If pipx installed via homebrew uses the |
|
The solution to this would be to convince the Homebrew people in some way to maintain a symlink for the Python major.minor version ( |
|
As virtualenv maintainer, I tend to disagree. I think the path forward here will be for pipx shim to automatically check if the home python changed (can just check the venvs pyenv.cfg home key), and if it did trigger a reinstall against the new Python (recreate venv + reinstall packages). @cs01? |
|
Here's an attempt at making this problem reproducible:
@gaborbernat are you sure its necessary to reinstall all packages for a minor python upgrade? At least in the homebrew context, I think we can make python minor upgrades more pleasant by linking to the correct place. I think your idea makes a lot of sense for major python upgrades though. |
|
Here is a python script I hacked together that will fix all your virtual environments after an upgrade. It seems to work for me, but mileage may vary, some stuff is hardcoded, etc. Use at your own risk: import os
from functools import lru_cache
from textwrap import dedent
def find_most_recent_homebrew_python(python_base):
dirs = [
item
for item in os.listdir(python_base)
if os.path.isdir(os.path.join(python_base, item))
]
print("dirs = {}".format(dirs))
path_to_versions = os.path.join(
python_base, sorted(dirs)[-1], "Frameworks", "Python.framework", "Versions"
)
current_version_link = os.path.join(path_to_versions, "Current")
current_version = os.path.realpath(current_version_link)
return os.path.join(path_to_versions, current_version)
@lru_cache(maxsize=None)
def find_location_under_dir(base, location):
print(f"Searching for location {location} under {base}")
for (root, dirs, files) in os.walk(base):
if location in files:
result = os.path.join(root, location)
print(f"Found file {location} under {base} at {result}")
return result
if location in dirs:
result = os.path.join(root, location)
print(f"Found directory {location} under {base} at {result}")
return result
print(f"Unable to find location {location} under {base}")
return False
def main():
venv_dir = os.path.expanduser("~/.virtualenvs")
python_base = "/usr/local/Cellar/python"
python_dir = find_most_recent_homebrew_python(python_base)
print("python_dir = {}".format(python_dir))
for (root, dirs, files) in os.walk(venv_dir):
# print("root = {}".format(root))
# print("dirs = {}".format(dirs))
# print("files = {}".format(files))
abs_files = [os.path.join(root, f) for f in files]
broken_links = [
f for f in abs_files if os.path.islink(f) and not os.path.exists(f)
]
for link in broken_links:
resolved_link = os.path.realpath(link)
print(f"Found broken link {link} pointing to {resolved_link}")
base, fname = os.path.split(link)
new_target = find_location_under_dir(python_dir, fname)
# If we found a new target, repoint the link
if new_target:
print(f"Repointing {link} to {new_target}")
os.remove(link)
os.symlink(new_target, link)
continue
# If we didn't, see if we can find the filename of the old link
missing_base, missing_fname = os.path.split(resolved_link)
new_target = find_location_under_dir(python_dir, missing_fname)
if new_target:
print(f"Repointing {link} to {new_target}")
os.remove(link)
os.symlink(new_target, link)
continue
# Require manual intervention here
else:
msg = dedent(
"""
Unable to fix broken link {link} pointing to {resolved_link}.
You may want to manually intervene here.
Press enter to continue:
"""
)
input(msg)
if __name__ == "__main__":
main() |
|
my simple and dirty workaround is force pipx to re-create its shared virtualenv by deleting it, and reinstall all packages: |
This worked perfectly, thank you! Just had to change my virtualenvs folder in:
From |
|
oh boy. a year after this issue is opened I see that there's been no progress to resolve it without hacky workarounds 😬 in some respects I'm glad because what I thought to be the problem has been confirmed here, but I'm also not gonna worry too much about trying to make this work. I think I don't use this tool enough to justify the effort to side step the problem every time I do a Python upgrade. I know that sounds kinda snarky, I don't actually mean it to, I'm just honestly not a big enough user of this tool to want to have to mess around like this, to solve a problem such as this. It's good to see there are a few different workarounds at different levels the community of users has provided though (for future travellers to utilize, including myself if I do decide this tool is indepensible to my day-to-day workflow) ❤️ |
I'm on mac, and homebrew upgrades my python all the time. All my pipx packages are easily fixed when this happens by: This workaround doesn't seem too hacky to me. |
When I run |
|
Thanks, @Integralist, this is a bug, but separate from the original issue here. Could you please start a new issue with the above comment? In addition, could you show what version of pipx you are using? (i.e. the output from It may be that your version of pipx is so old that you don't have many of the updates that made |
|
To add on to what @itsayellow said, could you re run the command with the |
|
I also need to |
I ran
just yesterday, and everything worked fine. Then I ran
which upgraded my system Python from 3.7.2 to 3.7.3. Now pipx no longer works:
I assume the problem is that when pipx creates the virtual environments, it resolves
/usr/local/bin/python3as a symbolic link, which is problematic since the destination of that symlink changes every time Homebrew upgrades Python (thus making the old virtualenv's Python executable no longer work). I have experienced this problem with Pip as well, but presumably pipx could wrap this behavior somehow to make it work, or at least display a useful error message. At the least, it would be nice to havepipx reinstall-all python3work properly.Workaround is something like the following:
and manually prune any broken symlinks from
~/.local/bin.The text was updated successfully, but these errors were encountered: