Proposal: make uv the documented and supported way to install opensafely-cli

[evansd]

Background

We currently distribute the OpenSAFELY user tooling via PyPI. Our users often aren't Python users, and are running a variety of different operating systems with different levels of administration privileges, and have therefore installed Python in a variety of different ways. To cope with this we do two things:

• we test the tooling on a range of Python version (currently 3.8–3.10);
• we ensure that the tooling vendors all its dependencies so it can be installed in any environment without risk of interaction with other packages in that environment.

This has some annoying consequences:

• our code can't use features from newer Python versions;
• we're limited to only using pure Python dependencies because only those admit cross-platform vendoring;
• updating dependencies is more faffy and difficult because of the vendoring (sometimes requiring manual surgery).

Even worse, because the CLI tools depends on the pipeline library (for parsing project.yaml files) and job-runner (for running jobs locally) these limitations are transitively imposed on those libraries as well.

And, despite all this pain on our side, the user installation story still isn't great here because users still have to wrestle with installing Python somehow.

We've considered a variety of more drastic solutions here (e.g. build and distribute our own application which bundles Python itself; rewrite everything in Go) but not made any progress so far.

How might uv help?

uv is a tool for installing and managing Python and Python packages:
https://docs.astral.sh/uv/

It has a reasonably nice installation story across all platforms relevant to us:
https://docs.astral.sh/uv/getting-started/installation/

Once uv is installed it can take care of installing the appropriate version of Python and then installing our tooling and its dependencies in an isolated environment. This means we can require a specific, up-to-date version of Python and no longer need to vendor our dependencies.

Compared to the status quo there are literally no downsides. But there will inevitably be a switching cost in getting existing users to move across, and there are still some unanswered questions.

Unanswered questions

1. Is uv actually easy to install in the contexts are users are operating in?
   I think the first thing we should do is get some friendly users (spread across a variety of platforms) to try to install it, based purely on the docs uv provide.

2. Do we want to use uvx or uv tool install?
   The latter creates a persistent Python environment and puts the installed binary on the path; the former manages ephemeral environments behind the scenes. I think I'm inclined towards persistent installations but open to hearing views.

3. How do we ensure that all dependencies have pre-built wheels for all platforms we want to support?
   Although uv makes having compiled dependencies possible we still need to make sure that there exists e.g. a macOS-arm64 wheel for each of them. We'd want to enforce this in CI for every project that's upstream of opensafely-cli. I'm sure that's possible, but I don't think it's something we've tried to do before.

4. How do we handle updates?
   At the moment we go through various forms of hell in order to make opensafely upgrade invoke pip and upgrade itself. Maybe we could abandon that and just have the upgrade prompt print the relevant uv instructions.

5. How do we handle the switchover?
   At some point we'll have a version of opensafely-cli which should only be installed via uv and should not be pip-upgraded to from an old style installation. Can the tool itself reliably detect whether it's been installed via uv or not? If so then we can have it refuse to update itself and instead link to docs about how to switch to uv.