Distribute binary version that doesn't require a system python

[bloodearnest]

We currently distribute opensafely-cli via pip. This is not great:

1. it requires a user to have a system python installed
2. it means we need to support a wider range of pythons (which means job-runner needs to also support the same range)
3. it requires us to vendor all our dependencies to simplify the the UX, which increases maintenance costs, and prevent any C-level dependencies.

Ideally, we would distribute opensafely-cli as a binary, which:

a) includes the specific version of python we want to use
b) includes all the dependencies at build time
c) is platform specific so we can use C dependencies

One promising avenue for this is pyoxidizer. This is a rust tool chain that builds a platform specific binary with your code, deps, and a python interpreter (with only the parts of stdlib that your code uses).

[sebbacon]

See prior discussion here, including some possibly-out-of-date discussion on:

• signing packages
• slow startup times of pyinstaller
• a nice summary of installer options as they existed at the time
• is M1 supported?
• is it better to focus our energy on gitpod (or probably, now, Code Spaces)?

[bloodearnest]

is M1 supported?

Rust has Tier 2 support for M1 currently: https://doc.rust-lang.org/beta/rustc/platform-support.html. AIUI that means it should work. Python is reported to work on M1 too, but there's no official build.

However, an x86 build should run just fine on the M1, AIUI. We're not doing anything complex in opensafely-cli itself - that's all off boarded to docker.

is it better to focus our energy on gitpod (or probably, now, Code Spaces)?

Yes, in terms of prioritisation.

But unless we move to only supporting gitpod/Code Spaces, this work is still useful.

Currently, job-runner python code base has to maintain compatibility with python 3.8 and 3.9 (we should really add 3.10 and soon 3.11 to that) to support installing it on whatever version of system python users have installed.

Fixing this ticket would allow job-runner and opensafely-cli codebases to just support a single python version, and use that in prod and in opensafely-cli.

[ghickman]

Python is reported to work on M1 too, but there's no official build.

3.9.1 (released 7th December 2020) brought official support for Apple Silicon (that's M1 and M2 chips) via the universal2 builds of the runtime.

[ghickman]

This would also solve for the broken-on-macos update command.

This invocation of pip is incompatible with our documented method of installing opensafely-cli on macOS.

[bloodearnest]

I think we should not use pipx in the docs. It adds no value to the specific case of opensafely-cli, which has no dependencies by design and thus does not need to be installed in an isolated venv anyway.

[ghickman]

Very good point! Those docs predate our vendoring of dependencies, which is why we went with pipx. Captured in opensafely/documentation#1026