ENH: Create arm wheels for PyPI

[mattip]

It turns out there is now a free service at shippable for open-source packages to run on AARM64 hardware. NumPy has been experimenting with adding that architecture to our build matrix. Would there be interest in integratiing this into cython so we could get arm wheels on PyPI? The work on NumPy is numpy/numpy#11702. @jjhelmus is also trying this on Scipy and has a yml file that may be of interest.

[tylerjereddy]

+1, this would be extremely useful for us -- let me know if there's something I can do to help

[scoder]

More testing on other platforms is always nice. If it doesn't cost us too much in terms of maintenance, and gets us ready-made wheels for PyPI … that are actually useful for end users, that's enough of an advantage.

I'm happy to receive a PR.

[tylerjereddy]

@scoder Can you direct me to the appropriate resource to get started on the ARMv8 wheel generation process for Cython? It looks like there's a github repo: https://github.com/MacPython/cython-wheels

Should I start with that repo as a resource?

[scoder]

Integrating the arm build into the cython-wheels project would be nice, yes.

For the build steps, you can also look at the travis.yml script. Basically, we want to compile Cython (preferably in parallel, e.g. setup.py build_ext -i -j4), then build a wheel, then run the tests (see make test, but preferably also in parallel with -j). Finally, upload to the same place as the wheel builds, if possible.

Our travis CI setup (not the wheels build) uses separate build jobs for the C and C++ tests through a build matrix, which leads to a much faster turn-over, but for the wheel builds, we'd have to take care that we don't upload twice if we enable that. Also, disabling C compiler optimisations (CFLAGS="-O0") when running the tests speeds them up a lot.

[jjhelmus]

I do not believe PyPI allows the upload of armv8/aarch64 wheels. PyPI does allow armv6 and armv7 wheels although there is no standardized ABI compatibility for these and therefore it may not be a good idea to upload them, see pypi/warehouse#3668.

[scoder]

Ah, right. PyPI is very restrictive when it comes to Linux binaries, and for good reason. They pretty much only allow manylinux builds for x86/amd64, since those tend to work on most Linux systems.

That makes ARM wheels appear a lot less interesting. I mean, we could still ship them e.g. from github releases, but they won't be found by pip unless you pass it the exact file URL.

[tylerjereddy]

Ok, so we'll likely just have to rely on either:

• caching the wheel in the CI and paying the ~13 minute build time cost when pip grabs a new release of Cython (or when we change a pinned version)
• occasionally doing a manual build of the wheel and making it available at a specific URL as @jjhelmus has done recently

[jakirkham]

What CI are you using that has ARM support? Does it work without virtualization?

[tylerjereddy]

Shippable is the CI -- it is genuine ARM architecture if that's what you mean yes.

You can see the latest build result here for example: https://app.shippable.com/github/tylerjereddy/numpy/runs/56/summary/console

That also links back to the yml file I've been using too if you click on the commit hash.

[insertinterestingnamehere]

The --no-cython-compile option is pretty good too. I think you have to compile quite a few Cython files to see a significant benefit from compiling Cython.

[jakirkham]

Thanks for the info. Is that only ARMv8-A or do they support older processors as well?

[tylerjereddy]

Looks like v8 only; we could request access to 32 bit v8 as well, but that’s the only extra option I see. If there’s a compelling reason to need the older ones we could ask about it; they are pretty responsive so far.

…

On Wed, Aug 22, 2018 at 09:26 jakirkham ***@***.***> wrote: Thanks for the info. Is that only Armv8-A or do they support older processors as well? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2561 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHiXZoN3x5z-qN8lkM21PdPvdPnYwxAHks5uTYYvgaJpZM4WAw30> .

[vielmetti]

Very interested in seeing this progress! Also interested in doing the necessary standards work to get PyPI to handle arm64 binaries (realizing that's not an instant task).

[scoder]

We have ARM wheels since 0.29.22.
MacPython/cython-wheels@8a27562