pipx as a frozen standalone binary

[clbarnes]

pipx solves the problem of system tools depending on python installations, of which you may have one (a problem because messing with the system python is dicey), or many (a problem because you have to keep version-gardening).

Even if you're disciplined and keep as much as possible in virtualenvs, you still end up having to install one system tool outside of a virtual environment... pipx (the call is coming from inside the house!). If it would be welcome, I'm considering trying to freeze pipx into a static binary using PyOxidizer. It would then be pretty easy to deploy on different package managers etc. and generally be a "real" system tool, isolated from the system and other un-virtualenv'd pythons. Once pipx is installed, other python-based system tools are a breeze to install.

However, I suspect there might be some interaction between PyOxidizer and the bits of pipx which complain about being used in a virtualenv. I'll see what happens.

[clbarnes]

The way that the venv_metadata_inspector is currently read falls afoul of the resource loader issue documented here. In python 3.7+, this can be resolved with the ResourceLoader API.

[cs01]

the call is coming from inside the house!

😆

pipsi solved this by bootstrapping itself, and pipx used to install itself this way. I didn't really see a big benefit from doing this though so I stopped doing it with pipx because it added a bunch of extra confusion to people trying to install it for the first time. At that time pipx had I think zero dependencies and now it has a couple plus several transient dependencies. @mattsb42 made a clever package on PyPI called pipipxx which does this (from its README):

builds a temporary virtual environment, installs pipx there, and then uses that pipx to install pipx in your user local space, just like any other pipx-installed tool. What you end up with is a pipx installation that is itself managed by pipx.

Anyway, back to PyOxidizer:

The way that the venv_metadata_inspector is currently read falls afoul of the resource loader issue documented here. In python 3.7+, this can be resolved with the ResourceLoader API.

This isn't a problem is it? PyOxidizer bundles the entire Python runtime, so if the PyOxidizer version of pipx is built with 3.7, the resource loader api will be available and work just fine.

[cs01]

Oh btw this was discussed a little in another issue. #146 (comment)

[cs01]

If there are still difficulties locating or running the venv_metadata_inspector.py, you could also just embed it directly as a string in the code. That is how pipx runs it anyway, by passing it to python as a string (the -c option). The only reason it's in its own file is because it's easier to write it (autoformatting+linting) and test it that way.

pipx locates the file and reads it as a string:
https://github.com/pipxproject/pipx/blob/047a0be84ccfc70d40886eb9cfdef8b45bac5113/pipx/Venv.py#L66

then runs it here with the -c option:
https://github.com/pipxproject/pipx/blob/047a0be84ccfc70d40886eb9cfdef8b45bac5113/pipx/util.py#L82

[mattsb42]

FWIW, I've found that using a pipx managed pipx actually solves these problems very nicely.

The original idea behind pipipxx (which while I still enjoy the name probably needs to be published under a different name for better discoverability and clearer pronunciation guides) was simply that...pipx is not special; it's just another pip-installable Python package that exports some executables. I use this every day on multiple computers, and I have found it has some really nice characteristics:

1. All of my pip-installed tools are managed the same way.
   • This means that if anything breaks, or is doing strange things, it's usually the same root cause. It also means that there is a single way to update all of them.
2. All of my pipx-installed tools link to a single Python (~/.local/pipx/venvs/pipx/bin/python).
   • This is either a feature or a bug, depending on how you look at it. Personally, in practice, I have found it to be more good than bad. Because of the issues explored in All virtualenvs break when Python version is upgraded #146 (more on that later), keeping the Python links pointing to the right place across interpreter upgrades can be complicated. I don't have a good answer to avoiding that problem yet, but having a single thing to fix (the bin in the pipx venv) makes things much simpler to manage.
   • This also has some neat side effects, such as letting me easily try out different interpretters without requiring reinstallation. Probably not a very widely valuable side effect, but occasionally useful.
3. Entirely empty system and user site-packages! :D
   • For a system that was installed since I started using pipx:

$ python3 -m pip freeze
$

Now, that said, pipipxx is not without its own issues. There are some UX issues with the current install process that I need to sort out[1], and it has the same problems identified in #146 for at least the initial installation[2][3].

It also turns out that testing things involving ephemeral venvs is hard. :) Most of my usual toolset is built around testing inside venvs, and I can't do that with pipipxx.

[1] mattsb42-meta/pipx-in-pipx#4
[2] mattsb42-meta/pipx-in-pipx#10
[3] mattsb42-meta/pipx-in-pipx#11

[cs01]

Thanks for the write up @mattsb42! pipipxx and a PyOxidizer-pipx both seem useful to me.

Agreed about the name and some UX issues during installation. I recently installed it and it looked like it failed (a bare exception and Failed building wheel for pipipxx), but it didn't; everything worked fine! Also have no idea how to spell it. I just copy and paste it 😆 .

[mattsb42]

Yup, that was probably mattsb42-meta/pipx-in-pipx#4.

It was meant to be pronounced "pipx in pipx"...thus the name "pipipxx" with "pipx" replacing the second "p" in "pipx". Unfortunately, I have to explain that to each new person I encounter, so I'm pretty sure the name was too "clever" for its own good. :/

[jayvdb]

I've been doing a bit of analysis of 'fat binary' tools, which can be seen at https://github.com/storyscript/community/issues/8#issuecomment-540956151

PyOxidizer is great, but any use of __file__ breaks it, but all of the alternatives of __file__ are also broken. See indygreg/PyOxidizer#69 (comment)

https://github.com/Nuitka/Nuitka/ and PyInstaller have mostly solved the __file__ problem. voc and https://github.com/beeware/briefcase may be suitable if the Python 3 syntax here and in deps is 3.5-ish - there are a few bits it doesnt support.

Another option is to vendor everything, and I highly recommend https://github.com/dephell/dephell for doing vendoring automagically.

[AlJohri]

@cs01 can you tag this as a good first issue for someone to dive into? I think this would solve a ton of problems.

[gaborbernat]

I would not say this to be good as the first issue as can expose quite a lot of edge cases. Though I think here we can just do zipapp, I've done something similar on virtualenv which I think we can reuse.

[AlJohri]

Fair enough- it just seemed like a self-contained and discrete task but I of course haven't gotten my hands dirty with it yet :)

Do you want to link the example you've done before?

[gaborbernat]

pypa/virtualenv#1491

[gaborbernat]

To be fair given we only want python3.6+ might be enough to do a https://github.com/linkedin/shiv

[AlJohri]

I've got a version using PyOxidizer half way done.

pipx list works. Just need to solve the usage of sys.executable -m venv for creating venvs: indygreg/PyOxidizer#218

[gaborbernat]

IMHO that's the wrong abstraction level. PyOxidizer is meant to be a binary executable. It's not meant to create virtual environments. I'm fairly certain that creating a venv from within a pyoxidizer binary would not work.

[AlJohri]

@gaborbernat I think you're right. PyOxidizer cannot create venvs yet. I linked several issues within indygreg/PyOxidizer#218 detailing how this might be possible in the future.

In the mean time, I think the problem with zipapps is that they are still beholden to the system python installation meaning pipx will/can break when the global python is upgraded

What do you think of this approach for at least macos/linux: https://www.scylladb.com/2019/02/14/the-complex-path-for-a-simple-portable-python-interpreter-or-snakes-on-a-data-plane/

I think it would be preferable to just link to a stable, globally insalled python, in the correct way such that venvs don't break for minor upgrades. But the approach in the scylladb would theoretically shield even against major python upgrades.

[gaborbernat]

pipx is based on the global python by design. Removing this would imply we ship our own python for every platform... something I'm not sure we want to get into, as seems trying to do more than we can chew.

[uranusjr]

I played around publishing pipx to Scoop with the [Python embeddable package](The embeddable package).

https://github.com/uranusjr/pipx-standalone

scoop bucket add pipx https://github.com/uranusjr/pipx-standalone.git
scoop install pipx

(Edit: I merged to 32 and 64 bit manifests.)

Most things seem to work OK if I pass --python because the embeddable package does not contain venv (and I don’t think it can). I’m wondering what is the best approach to deal with this. @cs01 would you be open to provide a way to change the DEFAULT_PYTHON constant? (e.g. reading from environment variable PIPX_DEFAULT_PYTHON)

[gaborbernat]

I'm a bit on the edge with this; if the user needs to pass in a python why not just ship it as to be called via that python on start? Why complicate with scoop the runtime and venv python separations?

[uranusjr]

I would strongly prefer not depending on the packager-managed Python, since that means I’ll need to patch the pipx manifest when Python upgrades (see Homebrew’s plan to release Python 3.8; this happens every time Python releases a new version). I can help develop and maintain a recipe, but certainly don’t have the capacity to constantly follow a moving target to avoid breakage.

[yajo]

I managed to workaround this problem using just pipx and venv:

➤ docker container run --rm -it python:3.6-slim-buster bash

root@10651e70f23e:/# python3 -m venv venv

root@10651e70f23e:/# source venv/bin/activate

(venv) root@10651e70f23e:/# pip install pipx
Collecting pipx
  Downloading https://files.pythonhosted.org/packages/b7/63/f8a8119ff7abd5833c41ec0d7c0d46502462aab9a7bb649502723127289a/pipx-0.15.4.0-py3-none-any.whl
Collecting argcomplete<2.0,>=1.9.4 (from pipx)
  Downloading https://files.pythonhosted.org/packages/82/7d/455e149c28c320044cb763c23af375bd77d52baca041f611f5c2b4865cf4/argcomplete-1.11.1-py2.py3-none-any.whl
Collecting userpath (from pipx)
  Downloading https://files.pythonhosted.org/packages/d8/1e/bad15e27cf1190f24d837205aaddfe855ef920bb6ab162e3df3506c51e94/userpath-1.4.0-py2.py3-none-any.whl
Collecting importlib-metadata<2,>=0.23; python_version == "3.6" (from argcomplete<2.0,>=1.9.4->pipx)
  Downloading https://files.pythonhosted.org/packages/ad/e4/891bfcaf868ccabc619942f27940c77a8a4b45fd8367098955bb7e152fb1/importlib_metadata-1.6.0-py2.py3-none-any.whl
Collecting distro; platform_system == "Linux" (from userpath->pipx)
  Downloading https://files.pythonhosted.org/packages/25/b7/b3c4270a11414cb22c6352ebc7a83aaa3712043be29daa05018fd5a5c956/distro-1.5.0-py2.py3-none-any.whl
Collecting click (from userpath->pipx)
  Downloading https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl (82kB)
    100% |████████████████████████████████| 92kB 6.9MB/s 
Collecting zipp>=0.5 (from importlib-metadata<2,>=0.23; python_version == "3.6"->argcomplete<2.0,>=1.9.4->pipx)
  Downloading https://files.pythonhosted.org/packages/b2/34/bfcb43cc0ba81f527bc4f40ef41ba2ff4080e047acb0586b56b3d017ace4/zipp-3.1.0-py3-none-any.whl
Installing collected packages: zipp, importlib-metadata, argcomplete, distro, click, userpath, pipx
Successfully installed argcomplete-1.11.1 click-7.1.2 distro-1.5.0 importlib-metadata-1.6.0 pipx-0.15.4.0 userpath-1.4.0 zipp-3.1.0
You are using pip version 18.1, however version 20.2b1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

(venv) root@10651e70f23e:/# pipx install --python /usr/local/bin/python3 pipx
⚠️  Note: pipx was already on your PATH at /venv/bin/pipx
  installed package pipx 0.15.4.0, Python 3.6.10
  These apps are now globally available
    - pipx
⚠️  Note: '/root/.local/bin' is not on your PATH environment variable. These apps will not be globally accessible until your PATH is updated. Run `pipx ensurepath` to automatically add it, or manually modify your PATH in your shell's config file (i.e. ~/.bashrc).
done! ✨ 🌟 ✨

(venv) root@10651e70f23e:/# pipx ensurepath
Success! Added /root/.local/bin to the PATH environment variable.
Consider adding shell completions for pipx. Run 'pipx completions' for instructions.

You likely need to open a new terminal or re-login for the changes to take effect. ✨ 🌟 ✨

(venv) root@10651e70f23e:/# deactivate

root@10651e70f23e:/# rm -Rf venv/

root@10651e70f23e:/# bash -c 'pipx list'
venvs are in /root/.local/pipx/venvs
apps are exposed on your $PATH at /root/.local/bin
   package pipx 0.15.4.0, Python 3.6.10
    - pipx

[mattsb42]

@yajo FYI, that is almost exactly what pipx-in-pipx does for you. :)

https://pipx-in-pipx.readthedocs.io/

[itsayellow]

I've used pyinstaller in the past to package a self-contained app, which is very mature. The one thing I remember about that is you need to build on every separate platform to make the apps for those platforms. But with every major OS now available to us using GitHub Actions we should be able to automate that wholly in GitHub.

[pfmoore]

As a simple solution, why not just ship a zipapp of pipx? I've been using a self-built zipapp for some time now and it works pretty well (the biggest issue is that I have to remember to manually update it).

Even if the ultimate goal is something more sophisticated, IMO an "official" zipapp would be a useful starting point. It could be published as a github release here - there are tools (for example scoop on Windows) which make it pretty easy to track new github releases and automate upgrades.

[tucked]

Potentially relevant: https://discuss.python.org/t/announce-pybi-and-posy/23021

[yozachar]

As a simple solution, why not just ship a zipapp of pipx? I've been using a self-built zipapp for some time now and it works pretty well (the biggest issue is that I have to remember to manually update it).

Even if the ultimate goal is something more sophisticated, IMO an "official" zipapp would be a useful starting point. It could be published as a github release here - there are tools (for example scoop on Windows) which make it pretty easy to track new github releases and automate upgrades.

asdf-pipx can track pipx updates on Linux and MacOS.

[dukecat0]

Another new option is https://github.com/ofek/pyapp.