Supporting arm64, aarch64, and other platforms.

[TomAugspurger]

There are some desires / efforts to support other platforms like arm64 / aarch64 (#30641). I might have missed it, but I don't know that we've had a real discussion on if / how we are going to support those.

I'm not against adding support for either of these, but we should do it deliberately. I think that us uploading wheels / conda packages implies some kind of official blessing that we support those platforms, and will fix bugs specific to those platforms when they arise. These platform specific issues are extremely hard for community-maintained open-source projects to manage (we've had enough issues with 32-bit Windows / Linux). Contributors and maintainers don't have access to hardware and are stuck trying to debug things through CI.

Let's use this issue to discuss

1. Do we want to support these two platforms? (I think yes to both)
2. What are the requirements for us to add a platform?

• Freely available CI infrastructure
• NumPy supports it (any other deps? Cython? Arrow at some point?)
• Some level of popularity? We drop Python versions based on popularity, but this seems hard to measure for platforms we don't already officially support.

3. We need a person / team who will champion platform-specific issues. We can ping that person or team when CI / packaging issues arise with that platform.

I think we can discuss this more on our monthly dev call next week.

cc @pandas-dev/pandas-core & @odidev and @ossdev07 in particular.

[jreback]

+1 on these arches, even though this would add to our CI + release burden. Do we have stats anywhere that shows 'popularity' anywhere? does numpy support these?

[TomAugspurger]

NumPy does aarrch64 stuff at https://github.com/numpy/numpy/blob/master/shippable.yml, but doesn't currently have wheels (numpy/numpy#14886).

Not sure about arm64.

[cphang99]

The libreML project builds SDKs for machine learning applications using the BuildStream integration tool.

We have recently gained capacity for arm64 builds in our CI infrastructure, e.g. https://gitlab.com/libreml/libreml/-/merge_requests/253 with the potential of adding other architectures if hardware permits. We already have the capacity to track new releases of pandas, and to test this in our CI, in an automated fashion.

We would be interested in supporting this effort whereever we can.

[TomAugspurger]

Thanks @cphang99, will take a look. FWIW, we're fairly heavily invested in multibuild / https://github.com/MacPython/pandas-wheels for our wheel building.

---

@mattip can you share the status on what's needed for NumPy to upload arm64 wheels to PyPI? Is it completely blocked by pypa/manylinux#542?

I also see that @odidev added support for arm64 to SciPy in scipy/scipy#11867 (cc @rgommers). I feel like there's opportunity for coordinating efforts around building wheels here (and perhaps you already are).

[mattip]

There is MacPython/numpy-wheels#74, which is blocked right now on a failing aarch64 test. That would be solved by being able to move past manylinux2014 (of which pypy/manylinux#542 is a part) or by simply skipping the test, numpy/numpy#16214.

We are trying to coordinate. In discussions, it is felt there is no need to require a numpy wheel before releasing a scipy wheel, since it is much easier to locally build numpy than to build scipy.

[rgommers]

NumPy is now unblocked, we should have wheels for 1.19.0 (branching soon) it looks like (thanks @mattip!).

I feel like there's opportunity for coordinating efforts around building wheels here (and perhaps you already are).

As far as I can tell, it's similar to other wheel production efforts - we rely on the same multibuild machinery, and adopt the same strategies for releasing wheels (without much formal coordination, just copy what works). Pushing manylinux2014 wheels to PyPI whenever you have them ready sounds great to me.

[odidev]

Numpy wheel for AArch64 is available now. Please have a look at this

[mattkanwisher]

A lot of machine vision is happening on the edge with Linux boxes. We are seeing hour+ installs of packages like pandas. Would be amazing to have aarch64 wheels

[mattip]

The work is taking place in this PR MacPython/pandas-wheels#99

[amerry]

Pandas 1.1.5 provided manylinux_aarch64 wheels for Pythons 3.6 through 3.8, which was great, thank you for that! The 1.2 release, however, doesn't include a manylinux_aarch64 wheel for Python 3.7, though - only for 3.8. Was this deliberate?

There is also no manylinux_aarch64 wheel for Python 3.9 (neither in 1.1.5 nor 1.2) - not sure whether this is deliberate either.

[simonjayhawkins]

Pandas 1.1.5 provided manylinux_aarch64 wheels for Pythons 3.6 through 3.8, which was great, thank you for that! The 1.2 release, however, doesn't include a manylinux_aarch64 wheel for Python 3.7, though - only for 3.8. Was this deliberate?

Thanks to @janaknat for the aarch support for 1.1.4 and 1.1.5

see MacPython/pandas-wheels#102 and MacPython/pandas-wheels#121

contributions and PRs most welcome.

[simonjayhawkins]

2. What are the requirements for us to add a platform?

• Freely available CI infrastructure

see #38943 and MacPython/scipy-wheels#106 (comment)

[rgommers]

Also note that conda-forge has 1.2.0 packages for aarch64 as well as osx-arm64 packages. They have structurally better support; if you're on ARM64 right now you should strongly consider using conda-forge.

[janaknat]

@simonjayhawkins I've got a PR (MacPython/pandas-wheels#123) for pandas aarch64 python 3.9 wheel. Any interest in supporting it?

[RafayAK]

Can I help somehow fast-track this? Seems like no progress on this since last year

As @amerry has mentioned above Pandas aarch64 packages were being built for 1.1.5 release, why this no longer the case?

[pg1671]

Running on the T4g instances on AWS and compiling Pandas takes over half an hour which is more than half the time it takes for my chef script to setup the server.

Would be great if pandas had an aarch64 package as these AWS instances have big price performance advantages.

[jreback]

the blocker is CI testing; though we recently got some credits from travis for this

i don't think we have a problem doing the releases

if folks want to donate time to help out would be great

[mroeschke]

I think we've largely committed to supporting ARM at this point:

1. We test ARM via CircleCI: https://github.com/pandas-dev/pandas/blob/main/.circleci/config.yml
2. We build ARM wheels on via https://github.com/MacPython/pandas-wheels/blob/master/.travis.yml and hopefully cibuildwheel soon: BLD: Build wheels using cibuildwheel #48283

so I think we can close at this point and support ARM bugs as they crop up.