bindings: python: add a script to generate armv7l wheels #93
Conversation
|
Hey @brgl, here's a script that makes armv7l (32-bit ARM) Python wheels, which I have been using in my own projects. The script passes The first question is whether you think that it makes sense to add it to the libgpiod repo in order for you to produce armv7l wheels and upload them to PyPI when you create new releases. If so, my next question will be about testing. I have only tested the wheels on my own projects that make very basic use of libgpiod (reading and writing GPIO pins), but I see that the Python bindings have unit tests that can use a chip simulator. Anyway, one question at a time. :) |
|
I'm not opposed to this in principle, however I would like to reiterate the concerns from: The purpose of publishing wheels on PyPi is to provide wheels that are compatible for all boards for the architecture target. 32bit ARM wheels are not prolific in the ecosystem after all of these years probably because ensuring compatibility has been difficult. If your target is RPi, the wheels published on piwheels should generally be sufficient. They are purpose built for that platform and should be as easy as pointing to the index either via pip or your package manager. Wheels on piwheels are not cross platform compatible per pypa/manylinux#1405 (comment) Also, armv7l is one subset of the 32bit ARM architecture family. This does not account for armv6l or armv8l and does not account for any optionally licensed features such as NEON or VFP. armv6l with no optional extensions enabled should technically be compatible through armv8l. I'm not sure whether the python package resolvers would understand that or not. I believe most of the variance exists in how floating point is handled, so gpiod may not necessarily be affected, but it's important because building wheels has the chance of packing in linked libraries from the build host that are then referenced via -rpath. If those libraries are built with unsupported instructions and executed, it could crash... a problem i've definitely had with PPC and their 32bit architectures. You'd have to ensure that gpiod and dependent host libraries are not compiled with VFP, NEON/SIMD, etc since these are all optional addons to the core ARMv7 instruction set. An x86_64 analog would be things like AVX/AVX2/AVX512/SSE/SSE2 support This is not an issue with 64bit ARM because aarch64 is a common subset for all armv8 chips. It's akin to having specified one of the x86_64 psABIs where you can guarantee a subset of instructions are supported. The 32bit ARM descriptors do not encompass this. If there is still a taste for this, then I would suggest providing evidence that these wheels work across a suite of armv7 chips. What i don't want to have happen is we upload these and then have a support nightmare because they don't work on someone's esoteric chipset. I'm also not confident how Python's packaging utilities handle installing packages on 32bit arm and how well tested that is. it looks like Anyway, I think these are some of the things that need to be considered before accepting this PR. |
Add a 'generate_armv7l_wheels.sh' shell script that makes armv7l Python wheels for combinations of libc (glibc, musl) and selected CPython versions. Currently, Python wheels are generated for the x86_64 and aarch64 CPU architectures only, using the 'cibuildwheel' tool invoked by the existing 'generate_pypi_artifacts.sh' script. The 'cibuildwheel' tool, in turn, relies on the PyPA manylinux and musllinux Docker images that, sadly, are not available for the armv7l architecture that is still common in edge devices. It was previously suggested that Raspberry Pi users could rely on the piwheels.org repository for armv7l gpiod wheels. Alas, that repository lags significantly behind the latest CPython releases. It currently provides gpiod wheels for CPython versions 3.9 and 3.11 only, while CPython 3.12 was released a year ago and CPython 3.13 is about to be released. It turns out that armv7l Python wheels can be relatively easily generated without 'cibuildwheel', in about 160 lines of structured shell script that introduces no new dependencies (the script depends on Docker and binfmt_misc only). Instead of manylinux and musllinux Docker images, the script uses the official Python image on Docker Hub, which is available for armv7l in Debian and Alpine flavours (for glibc and musl libc respectively). Signed-off-by: Paulo Ferreira de Castro <[email protected]>
pdcastro
changed the title
pdcastro
force-pushed
the
armv7l-wheels
branch
from
6db5bd1
to
971f2bc
Compare
|
@vfazio, @brgl, thank you very much for your informative comments. I read all the links and learned several things. Refined goal PiWheels By the way, in some future issue / PR, we could investigate the possibility of How For example, given a wheel file named: ... Of interest, as expected, this means that a wheel tagged Soft vs. hard float
def _have_compatible_abi(executable: str, archs: Sequence[str]) -> bool:
if "armv7l" in archs:
return _is_linux_armhf(executable)Note the This means that $ docker run -it --pull always --platform linux/arm/v5 -v "${PWD}:/dist" \
--hostname bullseye-armv5 python:3.12.5-slim-bullseye bash
root@bullseye-armv5:/# uname -m
armv7l
root@bullseye-armv5:/# python -c 'import sysconfig; print(sysconfig.get_platform())'
linux-armv7l
# Below, the absence of 'hf' in 'gnueabi' indicates a soft-float distro/libc.
root@bullseye-armv5:/# ls -ld /lib/arm-linux-*
drwxr-xr-x 1 root root 4096 Sep 5 09:39 /lib/arm-linux-gnueabi
root@bullseye-armv5:/# pip install /dist/gpiod-2.2.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
ERROR: gpiod-2.2.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl is not a supported wheel on this platform.Effectively, A soft-float wheel cannot be used on a hard-float system
Given that The script provided in this PR, which uses the official Python image in Docker Hub for the
However, this may not be much of an issue in practice. Despite a fair amount of searching, I could not find a single Linux distro pre-built for musl libc and soft-float ARM v7 or below (v4, v5, v6, v7). I found a few musl-based Linux distros or distro “stages” pre-built for ARM v7 with hard float (VFP required), for example Alpine Linux and Gentoo’s Stage 3 Musl, but none for soft float. (The Alpine Linux Downloads page uses the term It was a similar story for Linux distro images on Docker Hub. The official The fact that I did not find any musl-based, soft-float ARM v7-or-below pre-built Linux distro only potentially proves that my search was not thorough. (By the way, I also went through the musl-libc.org list of projects using musl.) But if indeed such pre-built distros are so hard to find, I am tempted to conclude that:
These tentative points are also supported by the fact that most of the glibc-based, soft-float Linux distros that I came across targeted ARM v4 or v5 rather than v6 or v7. Consequences of publishing
Conclusion @vfazio, @brgl, what do you think? If we agree to proceed, I will then look at Bartosz’s earlier request of extending the existing script. I might propose having the existing script automatically call the new script but still having separate files, because the logic intersection is minimal and each script’s responsibilities distinct. The original script sits at a higher level of abstraction and does not deal with Docker directly; it orchestrates the call to |
|
Caveat lector: I'm not the maintainer of this repo so my thoughts/opinions/rationale should be discounted accordingly. I think there are a number of things that warrant discussion here should we go this direction, but I think the first step is identifying the use cases to justify the work and continued maintenance. Piwheels is purpose built for RPi wheels for those using RPi OS. It does have wheels for gpiod==2.2.1 https://www.piwheels.org/project/gpiod/ The reason Piwheels does not support newer CPython releases is because the Debian distro used by RPi OS only comes with a fixed version. For Bookworm based builds, it's 3.11. Debian Trixie is going to target 3.12 at which point I would expect 32bit wheels to be available there (assuming there is still a 32bit distro available). For RPi, the use cases where piwheels aren't supported/guaranteed to work:
For a subset of RPis, migrating to a 64bit OS and using the AArch64 wheels should work. I and members of my team have tested these wheels on:
The scenarios not covered by piwheels + AArch64 wheels are:
Note also from piwheels' own FAQ, emphasis mine:
I do not have hard numbers (I'm not sure how I'd even get them) but I do not envision the above edge cases account for a significant percentage of deployment scenarios... at least, IMO, not a significant enough percentage that requires this repo to shoulder the responsibility of building compliant wheels and demonstrating a cross-architecture support matrix for RPI and other ARM cores. Wheels can still be compiled on-the-fly via sdist. If the use case is significant enough, i imagine people deploying in these edge cases have other packages for which they may need wheels and may already have a private package index for this case. I actively uses the 64bit wheels in a project and have a vested interest in seeing that those continue to work, however, I do not have the time to help test/debug/maintain the armv7l builds. Given the blurb from the piwheels team and the testing/maintenance burden, I'm hesitant to pursue these without further evidence of need and someone dedicated to helping maintain them. |
|
That this is for sure an awesome investigation but this is way tl;dr for this week that's super loaded for me. I already promised @vfazio to review his patches on linux-gpio and am not any closer to doing it. Could I get a short summary of your results?
TBH I don't care about binary distributions enough to impose on myself strict limits on which C APIs to use. Comfort of coding will always take precedence over ABI compatibility issues for me. |
|
The use case My own current project environment is just one data point, but for what it is worth:
Generalizing the previous points:
Let me emphasize the last point. PiWheels’ choice of producing wheels for a couple of Python minor versions only (3.9, 3.11) really diminishes its reliability, even within the Pi ecosystem. Python versions other than 3.9 or 3.11 were described as “non-standard Python deployments”, but this merely means Python versions other than the one that ships with a particular OS release. When a developer or team makes a choice of programming language version, like Python 3.10, in my experience it is rarely based on the version that comes with an OS release. Instead, the choice is usually based on compatibility with existing source code or libraries required by the project. In the absence of source code or library restrictions, the latest stable Python version would be the preferred version, in order to make the most of the programming language’s new features and improvements. There are popular tools like pyenv that are dedicated to making it easy to install multiple versions of Python, and then there are Docker containers too! Re “wheels can still be compiled on-the-fly via sdist”, in a way it could be argued that Promoting
I thought that the last option would be the best, by far. After all, I had already written the script that builds the The Raspberry Pi is the poster child of GPIO programming with Python — with literally a bunch of GPIO pins sticking prominently out. It would not be too much of a stretch to say that the Pi’s mission is “to make GPIO accessible to all”! 😀 Meanwhile, the PiWheels’ FAQ
Indeed, here’s the output of $ docker run --rm -it --platform linux/arm/v7 -v "${PWD}:/dist" -w /dist python:3.13-bookworm bash
...
root@4ed72415f483:/dist/piwheels# readelf -A _ext.cpython-311-arm-linux-gnueabihf.so
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "6"
Tag_CPU_arch: v6
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-1
Tag_FP_arch: VFPv2
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_align_preserved: 8-byte, except leaf SP
Tag_ABI_enum_size: int
Tag_ABI_VFP_args: VFP registers
Tag_CPU_unaligned_access: v6With this context, it’s understandable that PiWheels gets defensive. However, wheels built with the script provided in this PR contain actual ARM v7 binaries built with root@4ed72415f483:/dist# readelf -A _ext.cpython-312-arm-linux-gnueabihf.so
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "7-A"
Tag_CPU_arch: v7
Tag_CPU_arch_profile: Application
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-2
Tag_FP_arch: VFPv3-D16
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_align_preserved: 8-byte, except leaf SP
Tag_ABI_enum_size: int
Tag_ABI_VFP_args: VFP registers
Tag_CPU_unaligned_access: v6The burden of testing and maintaining |
Sure, here it is:
No worries if you need some time to review this PR. Thanks again for your attention. 👍 |
Ok.
Well, I've seen my share of arm7 shenanigans to not take anyone's word for it. :)
I see the script generating wheels for up to v3.12. @vfazio are you going to update it for v3.13 as well? Or is it system-dependent (my system has python v3.12.3)?
Ok.
Yeah but then we're running in compatibility mode.
I'm not saying no but I will defer to @vfazio who did all the work on publishing the wheels so far.
The patch should still go through the |
There was a problem hiding this comment.
Why is it still a separate script? Can it not be integrated into the existing one for some reason?
There was a problem hiding this comment.
Let’s first make a decision on whether or not armv7l wheels will be published to PyPI.
There was a problem hiding this comment.
It's not super straight forward unfortunately. cibuildwheel does have hooks for specifying the docker container used to generate a specific set of wheels, however, they didn't include armv7l until recently (and even then, only partially) so it's not as simple as specifying CIBW_{MANY,MUSL}LINUX_ARMV7L_IMAGE= and kicking off cibuildwheel.
I think they, too, are deferring to piwheels to provide those images for glibc environments because RPi OS is pretty much the defacto glibc based distro.
To add this support, a custom docker container needs to be generated with requisite dependencies, such as pip/build/etc and then invoked, preferably following the steps that cibuildwheel does it:
https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/linux.py#L161
So, stage one would generate standard wheels via cibuildwheel then a second stage would have to construct the container to generate armv7l wheels. I don't think there's any reason it couldn't be folded into the current script.
I'm not sure if we want to reinvent the wheel (pun partially intended). It looks like some people have build containers available: https://github.com/bjia56/armv7l-wheel-builder.
There was a problem hiding this comment.
Oh wow, so two weeks ago cibuildwheel added support for musllinux armv7l wheels, but not manylinux. I didn’t see that one coming! 🙂
Re https://github.com/bjia56/armv7l-wheel-builder, their Dockerfile is building binutils and glibc from source, and targeting armv6. It’s much simpler and faster to use the official Python base image on Docker Hub, as I’ve done in this PR.
If we are happy in principle to have armv7l wheels published to PyPI, I’d be happy to amend the existing script to use cibuildwheel for musllinux, and to add a few script functions to cover the manylinux case, along the lines of the work I have already done in this PR.
@brgl 3.13 was released just a few days ago. We can certainly add 3.13 wheels. I think we'd skip free-threaded wheels for now, however, as I'm not up-to-date on that feature and what changes may be required from the bindings to play nice in a GIL-less build. Adding 3.13 wheels should be as simple as: diff --git a/bindings/python/generate_pypi_artifacts.sh b/bindings/python/generate_pypi_artifacts.sh
index c2fb79f..6385a8a 100755
--- a/bindings/python/generate_pypi_artifacts.sh
+++ b/bindings/python/generate_pypi_artifacts.sh
@@ -108,8 +108,7 @@ python3 -m "${venv_module}" .venv
venv_python="${temp_dir}/.venv/bin/python"
# Install build dependencies
-# cibuildwheel 2.18.1 pins the toolchain containers to 2024-05-13-0983f6f
-${venv_python} -m pip install build==1.2.1 cibuildwheel==2.18.1
+${venv_python} -m pip install build==1.2.2 cibuildwheel==2.21.3
LIBGPIOD_VERSION=${src_version} ${venv_python} -m build --sdist --outdir ./dist "${source_dir}"
sdist=$(find ./dist -name '*.tar.gz')
I don't necessarily want to be the decision maker here, I just want to make sure we're doing our due diligence and there's someone motivated and able to debug build/execution issues. As I mentioned before, I have a vested interest in the 64bit wheels as the company I work for is using these bindings to automate the testing of GPIO on boards we manufacture. Unfortunately, even things like #85, which I'm on the hook for, will be difficult for me to get around to due time and the foreign build environment (I haven't messed with Yocto in 5 years). I would obviously strongly prefer that there be evidence that the armv7l wheels are tested on another ARM core variant besides RPi (mango Pi, NXP arm core, etc) and demonstrated to work, even if it's done via a 32bit container with /dev/gpio* mounted through. Out of curiosity, i queried the platforms that are pulled gpiod 2.x in the past year. Note that this does not reflect scenarios where packages may be cached behind a pypi proxy like Sonatype Nexus:
|
Ha! Is there any writeup on this? I would definitely want to make sure the bindings to work fine without the GIL. |
https://peps.python.org/pep-0703/ https://docs.python.org/3/whatsnew/3.13.html#free-threaded-cpython |
@pdcastro Sorry if i'm coming off like an asshole, that's not my intent. I just want to make sure we're accounting for edge cases and not digging ourselves a maintenance nightmare. |
I am willing to accept this but |
Isn't this expected? If you have a 64bit kernel, but a 32bit userspace, then we would expect pip to install 32bit wheels linked against 32bit libc? Or am i misunderstanding the situation? |
Is that when it happens? I'm not really sure what |
Thanks for these numbers, @vfazio 👍 Wow, I expected that Thank you both for the review. |
|
And I am honestly suprised by the tens of thousands of downloads in general! |
I’ve already closed the PR as per previous comment but just for completeness: I meant a scenario where |
|
Not to necropost but i wanted to explain a bit. You can run your own queries off of https://packaging.python.org/en/latest/guides/analyzing-pypi-package-downloads/#useful-queries I think the reality is that for RPis, a lot of people use RPi.GPIO or pigpio which do not use the gpiochip character device and avoid the claim system altogether by simply memmapping the GPIO registers or link against c libs like libbcm2835 which essentially do the same. This probably accounts for some of why 32bit arm downloads arent a high percentage of downloads. It wasnt my intent to shut down this MR, I just wanted to provide some numbers. Maybe we can rerun the query in a few months to see if this has picked up. I do think we could press on the cibuildwheel group about getting a stable manylinux image to build within now that there is one for musllinux. |
|
Just so i don't have to remember how i wrote the queries: SELECT details.cpu , COUNT(*) AS num_downloads FROM `bigquery-public-data.pypi.file_downloads` WHERE file.project = 'gpiod' AND
DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 360 DAY) AND CURRENT_DATE() and
file.version LIKE "2.%"
GROUP BY details.cpu
ORDER BY `num_downloads` DESCSELECT file.version, details.cpu , COUNT(*) AS num_downloads FROM `bigquery-public-data.pypi.file_downloads` WHERE file.project = 'gpiod' AND
DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 360 DAY) AND CURRENT_DATE() and
file.version LIKE "2.%"
GROUP BY file.version, details.cpu
ORDER BY file.version DESC, `num_downloads` DESC |
Add a
generate_armv7l_wheels.shshell script that makes armv7l Python wheels forcombinations of libc (glibc, musl) and selected CPython versions.
Currently, Python wheels are generated for the x86_64 and aarch64 CPU architectures only, using the
cibuildwheeltool invoked by the existinggenerate_pypi_artifacts.shscript. Thecibuildwheeltool, in turn, relies on the PyPA manylinux and musllinux Docker images that, sadly, are not available for the armv7l architecture that is still common in edge devices.It was previously suggested that Raspberry Pi users could rely on the piwheels.org repository for armv7l gpiod wheels. Alas, that repository lags significantly behind the latest CPython releases. It currently provides gpiod wheels for CPython versions 3.9 and 3.11 only, while CPython 3.12 was released a year ago and CPython 3.13 is about to be released.
It turns out that armv7l Python wheels can be relatively easily generated without
cibuildwheel, in about 160 lines of structured shell script that introduces no new dependencies (the script depends on Docker and binfmt_misc only). Instead of manylinux and musllinux Docker images, the script uses the official Python image on Docker Hub, which is available for armv7l in Debian and Alpine flavours (for glibc and musl libc respectively).This PR introduces such a shell script.
Sample usage:
(The
LIBGPIOD_VERSIONenv var is only needed if the SDist file hasn’t been created yet.)