Distribute buildstream Wheel packages that contain BuildBox

[ssssam]

Intro

Bst2 is available in PyPI, but only as a source distribution. Many packages on PyPI can be installed simply with pip3 install $package. Installing BuildStream is not so simple:

• A working cc or gcc must be available, along with CPython headers and libraries, so Cython modules can be compiled to binaries
• BuildBox must be installed, this currently implies building from source (see: buildbox issue about releasing binaries)

The Python Wheel format can be used to bundle and distribute binaries for our target platforms. When you run pip3 install buildstream and a suitable buildstream-*.whl is available on PyPI, the install should work even in a 'stock' container image:

• Cython modules are prebuilt so GCC isn't needed
• BuildBox binaries can be bundled in the wheel and installed in '$prefix/bin/' by Pip

Goals

The goal is a way to install BuildStream 2 in a single command on a wide range of machines, assuming:

• the OS doesn't provide its own BuildStream package (we prefer distro packages where available)
• the user trusts PyPI and the BuildStream release team

Non goals:

• replace distro builds. Anyone redistributing BuildStream should use the sdist or Git repo and build from source.

What about Docker?

The Bst 1.6 install instructions recommend running in Docker via bst-here. The Bst 1.9.95 install instructions also link to the buildstream-docker-images instructions. (At time of writing the buildstream.build website doesn't mention Docker directly).

Provisioning Bst via a container image is sensible for many situations and we will always ensure it is possible. Recommending it as the best way to use BuildStream is more nuanced.

Advantages of recommending users obtain Bst via Docker:

• Single-command setup that "just works" for most cases
• We already build and publish usable Docker images at https://gitlab.com/BuildStream/buildstream-docker-images

Disadvantages of recommending users obtain Bst via Docker:

• Dependency on a container host (we use hub.docker.io at the moment, but could use Github container hosting)
• Requirement that user can start a container - not easy if they're already in a container, or don't have the necessary privileges
• Builds have different requirements for host tools depending on what plugins are used, so there's no 'one size fits all' container
• Most build tools don't require a separate container to run, so this makes bst look like a 'heavyweight' tool
• Docker images are larger than wheels

[ssssam]

Steps for "first cut"

• Try cibuildwheel locally - done, config in pyproject.toml in sam/fat-wheels branch
• Try cibuildwheel in GitHub Actions - done, example
• Automate releasing wheels to GitHub and PyPI - ~done
  • Example implementation here
  • Example PyPI release: https://test.pypi.org/project/BuildStream/1.9.666.dev3/#files
• Include BuildBox binaries in wheel
  • Depends: https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/issues/1 (release BuildBox binaries)
• Ensure Bst uses the shipped buildbox binaries

Additional tasks

• Investigate if we can and should strip debug info
  • See: Strip debug symbols of wheels pypa/cibuildwheel#331
  • Difference seems to be: 2.5MB (-g) vs. 1MB (-g0)
  • With the addition of BuildBox binaries the wheel is up to about 20MB (example), the buildbox binaries are already stripped. So the Cython debug info is a small % of the total and not worth the worry.
• Investigate if we can build wheels for aarch64, riscv
  • See https://cibuildwheel.readthedocs.io/en/stable/faq/#emulation
  • We would need buildbox-integration to produce wheels for these architectures, there are no plans for that right now.
• Auto updates of cibuildwheel version
  • See https://cibuildwheel.readthedocs.io/en/stable/faq/#automatic-updates
  • We cannot use this because the thirdparty cibuildwheel action is not permitted on Apache runners
• Revisit musl support
  • If buildbox binaries don't support musl, then don't ship wheels for musllinux platform either.
  • buildbox-integration is building binaries that dynamically link to glibc 2.28. there are no plans to build separate musl binaries right now.

Verification

• buildstream-2.0.0-....whl can be installed and used in a venv, with bst 1.x installed on the host
• buildstream-2.0.0.....whl can be installed on the host, with bst 1.x in a venv
• Wheel can be installed and used on a stock container image of Ubuntu, Fedora, Debian without needing extra packages
• Test each wheel that we release on the platform it expects to run (e.g. the python3.7 wheel on an OS with python3.7)

[ssssam]

Some progress on this in https://github.com/ssssam/buildstream/tree/sam/fat-wheels:

• using bundled BuildBox binaries works in my limited testing
• bundling the binaries into .whl packages works when calling setup.py directly, but not when using pip or build to produce a wheel. More investigation needed into why these tools all behave differently.

[gtristan]

Note that this appears to be related to long standing epic #332