Re-enable linux ARM builds for CPython and numpy #1386
Conversation
|
Thanks!! Do you need linux-armhf?? If not, please leave it disabled. If linux-ppc64le builds though, let's enable it. And let's also see if scipy works with that too? |
|
Yes, we do require armhf. There are a wide variety of single board computers still out there that only support armhf. For example, the raspberry pi 2, and any raspberry pi 3 or 4 still running the older 32 bit OS. As for ppc64el and scipy, I can enable those and attempt to test cross compilation on my local machine, if I can get them working I'll push here |
|
I checked the failing workflows for cpython, that ipv6 error was one I had on my local system. I'll push the fix for it in a bit |
Right, but like you say they are old boards. What's wrong with using CPython 3.10? |
That's what we're doing right now. We depend on many other Javacpp bindings and require the latest versions where possible. On the Python side, we will be using bleeding edge and experimental packages. So being locked into a particular version of the cpython binding is concerning for us. If we are locked to 3.10 bindings and an incompatibility crops up, we would be forced to either be left behind or leave a significant portion of our users behind. While it might seem odd, we do have a very large number of older armhf boards in active use by our community, including the project leader, so abandoning them is just not an option. If armhf is a deal breaker, I understand, but our project would have to consider other options. |
|
ipv6 fix should be pushed. There were two other issues I experienced in my local build, requiring the following additional configure options:
I have not looked too closely into what these do, so I'm hesitant to push them until I know for certain the workflow builds fail without them. I'm checking scipy armhf now; it's getting late here, so I'm going to have to check arm64 scipy and ppc64le cpython/numpy/scipy tomorrow |
|
Well, you're the first user in years to mention the need for 32-bit builds, and since I'm not getting paid to do this, and no one is helping me, I have to cut somewhere, and that seems like a good place to start. You're going to need to get rid of your 32-bit boards eventually in a year or two anyway, so you know why not start the transition now before you're cornered by the market? If you have resources to spare though, that's a different question. Send them over here to help! |
|
It's not up to us on whether or not to get rid of older boards; our project is a hobbyist robotics framework designed to be easy-to-use and flexible. Our users include schools, clubs, and individual tinkerers that may not be able to find or afford newer boards. We're not afraid of breaking backwards compatibility so long as there is an easy migration path, but eliminating support for armhf has no easy migration path beyond forcing our users to junk their perfectly working hardware. We will eventually remove support when the community itself has decided it's time, as we have done for 32 bit x86, but that time is not now. As I stated in the PR description, I am willing to maintain the armhf and arm64 cpython builds myself. If there is additional cost to supporting armhf beyond that please let me know and I'll see if it's something I can help with. |
|
The next release is probably going to be next year and enabling builds that I'm probably going to disable before that isn't something I'll be doing, sorry. You can easily publish builds from your own fork though. Maven Central Repository is a hassle, but with GitHub Packages, we can use a GitHub account and there doesn't seem to be any restrictions about the groupId we can use. Could you check that out? |
|
I don't think maintaining a separate fork is within our project scope, helping to maintain arm support in upstream was all the resource bandwidth we had available. We will stick with 3.10 for now and evaluate other options. I will remove armhf soon |
|
Well you asked me how you could help, and I think that making that "forking" process easy is one way you could help. I get a lot of requests for custom builds, like you want linux-armhf, but just for CPython and nothing else, someone else wants Alpine, but just for OpenCV and nothing else, headless builds for FFmpeg, but nothing else, etc, it never ends! But no one wants to pay. Making the process easy for users to create and publish their own builds seems like the way to go here. You don't want to help with this? |
|
I've disabled armhf and added the two fixes mentioned above. I attempted to get scipy to compile, but it has other issues. Spitting out errors that numpy isn't installed, it's a check done in their setup.py, the reason it errors out is because when cross-compiling, numpy is a different architecture than the host and it blows up trying to import it |
Do you have something in mind for that? The only real way I could see that being accomplished is to have examples for the Github Packages thing you mentioned above and then require users to fork the repo, make their changes to configurations, and publish to Github Packages. Which is fine and all, but I would be fearful of those downstream forks getting angry when an upstream change eventually breaks their configuration. |
|
I would be willing to publish bindings I use. At least cpython, cuda, openblas, maybe hdf5. I get requests once in a while for fixing upstream bugs but have the same problem Sam does. @AutonomicPerfectionist if your problem is only publishing we could probably do a different namespace with the changes and publish a bit more frequently while backporting to the presets for when sam publishes. |
|
Forgot to remove armhf from the redeploy jobs.
I might be interested, but I'm not sure I fully understand what you mean. Are you saying make a fork with the needed changes, publishing them on a faster cadence, and then try to upstream the changes when the next release is a bit closer? |
|
@AutonomicPerfectionist yeah I wouldn't really want to "fork" as much as "add the necessary changes and release more often" the goal would be to stay compatible but share the work with Sam to avoid burden on him and releasing when we want. I would prefer to properly push this to maven central. I can do that under my namespace. |
|
It seems there was an untimely breakage in Github Actions arm64 package installation: I saw the same issue in #1384 |
I might be open to that, what do you think of it @saudet? |
|
So, I've deployed for now release builds for linux-armhf and linux-arm64 against the last release with your fixes: That said, the next release is probably going to happen after CPython 3.12 gets released, and cross-compilation might break again at that point, so let's revisit all this once it gets release. For now, let's merge this PR in the cpython branch, alongside builds for macosx-arm64 from @nightscape, and enable linux-armhf if you wish. Then, if cross-compilation breaks with CPython 3.12 and there is no one to fix it at that time, just like there was no one to fix it when CPython 3.11 came out, I won't have to do anything but leave the branch to rot. Sounds good? As for builds outside of the release cycle here, you can do whatever you want on your fork, and as long as it doesn't break any existing builds, we can merge fixes of your own here of course, even for builds that do not happen here. You can use Maven Central Repository, but like I said the easiest thing to do is probably go publish to your own personal repository from GitHub Packages since it works with your GitHub account and you won't have to fiddle with different package names. |
Oh, thank you! Yes, that should satisfy our current needs. |
|
@AutonomicPerfectionist I've upgraded the presets to CPython 3.12. I will merge anything that you have that works for CPython 3.12, not CPython 3.11. However, if you do not provide an update to support ARM now, before the release, please do not expect them to be available after the release. |
|
@saudet thanks for the heads up! I've merged master into this branch but won't be able to test it until later. When is the expected release date? Edit: Apologies, I did not see you had pushed updates yourself, let me know if you want me to revert that merge commit |
In a couple of months probably
I don't think I pushed anything on your branch, I had just changed the target branch, and I've just changed it back to master, but it does look broken now. If you want to continue on this pull request, you'll probably want to rebase onto master |
I see what happened, Github put the commits you made to master above my comment since they're older so I misinterpreted that as being made to my branch before my comment. I did check out the workflows and the only failing ones so far appear to either be Mac-related or a failure to update Ubuntu packages. I'll check on it over the weekend to see if there's anything that needs to be changed in regards to that |
|
Please revert changes related to Mac, yes. |
AutonomicPerfectionist
force-pushed
the
cpython-numpy-linux-arm
branch
from
52c4b3d
to
0769c3d
Compare
|
I've rebased on master so we don't have to deal with the several merge commits I had done over the course of this PR. There is no longer anything Mac-related in here, but as a consequence, the commit IDs and timestamps have been reset. As for the Armhf builds we had discussed previously, I am fine with leaving them disabled for the next release. We still have users stuck on older hardware, but now that the supply of modern Raspberry Pis has stabilized, I believe it's fine now to ask for users to upgrade. |
|
Sounds good, but the build for NumPy fails with:
|
|
I've switched the numpy build to use |
|
From what I understand, it should work pretty much transparently with crossenv: |
|
Unfortunately that was only the case before numpy switched their build system from the standard |
|
Diving ever deeper, it seems you can re-enable the older setuptools-based build system for now. I found this by looking at what Termux does: Seems they left the old pyproject TOML in the repo and you can swap to it by renaming it. Testing this on my local machine resulted in a successful |
|
@saudet are there any outstanding issues with this PR? I believe it should be functional as is but I have not tested the resulting artifacts on arm hardware yet. Also, out of curiosity, what's the position of this project on RISC-V? I'm considering getting some hardware for myself to help port some of our dependencies so we're ready when it starts being used en masse. Just wondering if additional support PRs would be welcome for a later release (probably won't have hardware in time to meet this next release) |
distutils has been removed from CPython 3.12. It looks like the NumPy team has hacked it to make it work with CPython 3.12 and added to NumPy instead, but it doesn't work with dependencies like SciPy, so please make it work without distutils. This package is dead, I'm not going to deal with it.
If you can figure out how to make that cross compile too, sure, that sounds fine. |
|
The builds are still succeeding, so let's merge this for now for the release and I'll just disable it again when it breaks I guess. Merging this might give you more incentive to work on a long-term solution :) |
|
@AutonomicPerfectionist FYI, the build using distutils and setuptools is gone from NumPy 2.0.0 |
|
@saudet thanks for letting me know! Sorry I didn't respond to your previous message, Github failed to notify me about that one... I'll work on fixing the builds as soon as possible, however I am in the middle of extensive and devastating flooding, so I'm unsure of when numpy can be fixed. |
|
Got it, take care! |
This PR fixes an issue where numpy could not be cross-compiled to linux-armhf and linux-arm64. The issue was an incompatibility between
crossenv, a cross-compiling virtual environment package, and Python 11. A fix was merged to crossenv in benfogle/crossenv#104 and released as crossenv version 1.4. This PR updates the crossenv version to 1.4, allowing cross-compilation again. Thus, it also re-enables builds for linux-armhf and linux-arm64 for both numpy and its dependency cpython.Theoretically, this should also fix cross-compilation for ppc64le, but I have no way of testing that platform, so I have opted not to re-enable builds for it.
Why is this a draft?
On my local system, cpython required additional changes to the cppbuild.sh script to successfully cross-compile. I would like to verify that those changes are not just related to my local configuration before I push them.
Additionally, I spent a while researching Python cross-compilation, and I would like to compile what I've learned into a
CROSS-COMPILATION.mdinformation guide for anyone who stumbles into the same issues in the future.Finally, I need to verify that cross-compiled builds don't have some fundamental problem with them even after compiling successfully. That will be verified on an armhf 32 bit raspberry pi by another member of my group.
Future Maintenance
I am willing to maintain the cpython linux-armhf and linux-arm64 builds as long as I can. If there are any future issues with them, please ping me in a new issue here on Github and I will respond as soon as possible. If the issue is serious enough or inhibits builds in other parts of the project, these arm builds may be disabled once more until I can fix them, if possible.
I do not intend to be the primary maintainer of numpy arm builds. Unfortunately, I have no use case for them and no way to test them myself. If they break and there is no one else to fix them, it is likely they will be disabled again.
Additional Information
See #1381 for more information.