[Python] Build wheels for manylinux2010 tag

[asfimport]

There is now work in progress on an updated manylinux tag based on CentOS6. We should provide wheels for this tag and the old manylinux1 tag for one release and then switch to the new tag in the release afterwards. This should enable us also to raise the minimum compiler requirement to gcc 4.9 (or higher once conda-forge has migrated to a newer compiler).

The relevant PEP is https://www.python.org/dev/peps/pep-0571/

Reporter: Uwe Korn / @xhochy
Assignee: Antoine Pitrou / @pitrou

Related issues:

• [C++] Consider adding Abseil (Google C++11 standard library extensions) to toolchain (blocks)

PRs and other links:

• GitHub Pull Request #4391
• GitHub Pull Request #4675

_{Note: This issue was originally created as ARROW-2461. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Turns out glibc 2.12, which is the baseline in manylinux2010 but not manylinux1, is required by the Abseil C++ library.

[asfimport]

Wes McKinney / @wesm:
Oy. So RHEL 6 has glibc 2.12 https://distrowatch.com/table.php?distribution=redhat

The base image for manylinux1 is centos5 which tracks RHEL 5.

https://distrowatch.com/table.php?distribution=centos

I don't know if this is a radical proposal but we should promote our manylinux1 image to be based on centos6, which I guess is the same thing as manylinux2010. PyTorch and TensorFlow have already created toolchain problems – not to say that we should make things worse – but I am not sure that this would do very much harm compared with, say, using a different compiler (like Clang with statically linked libstdc++ instead of using the RedHat devtoolset)

I don't know if we are doing ourselves any favors by being stuck on centos5 glibc, and the impacted users may be slow-to-upgrade enterprises which, I would hazard a guess, aren't contributing to Apache Arrow.

[asfimport]

Wes McKinney / @wesm:
According to https://access.redhat.com/support/policy/updates/errata RedHat has retired RHEL5 and is continuing extended support to those paying for it through end of 2020. I expect these cases to be quite exceptional and not reflecting of the average Arrow adopter or even Python user

[asfimport]

Antoine Pitrou / @pitrou:
I'm not sure what you're proposing here. The manylinux1 PEP requires that we don't rely on anything younger than glibc 2.5. So using Abseil in a manylinux1 wheel is out of question (unless we want to be a bad citizen like Tensorflow is).

Note that manylinux2010 is based on CentOS 6, i.e. its compatibility requirements for base libraries are based on what CentOS 6 provides.

[asfimport]

Wes McKinney / @wesm:
So I guess my main question is: If we stop shipping manylinux1 wheels in favor of manylinux2010, will that disrupt users, or will it "just work"? Are there other examples of projects using manylinux2010? AFAIK the only thing "bad" about using the manylinux1 tag on a wheel built on centos6 would be the minimum glibc version. So while still bad I think it's significantly less bad than what TensorFlow is doing, which is using a non-conforming compile and link procedure, in addition AFAIK to building on a newer Linux image. The only people impacted by this change would be users on very old Linux

[asfimport]

Antoine Pitrou / @pitrou:
Ha, apparently pip may still not have support for manylinux2010:
pypa/pip#5410

As for compatibility, the PEP says:

manylinux1 wheels are considered manylinux2010 wheels. A pip that recognizes the manylinux2010 platform tag will thus install manylinux1 wheels for manylinux2010 platforms – even when explicitly set – when no manylinux2010 wheels are available.

[asfimport]

Antoine Pitrou / @pitrou:

AFAIK the only thing "bad" about using the manylinux1 tag on a wheel built on centos6 would be the minimum glibc version.

Do we link libstdc++ statically? If so, hopefully that would be the only thing, yes.

[asfimport]

Wes McKinney / @wesm:
My understanding is that the libstdc++ details are all handled by RedHat's devtoolset toolchain. So I think what symbols are statically linked and binary compatibility should be the same (save for the glibc issue) whether the build is run on centos5 or centos6

[asfimport]

Uwe Korn / @xhochy:
manylinux2010 is sadly not yet finished. Once it's ready, I'm happy to switch to it. This will though require pip users to update to the latest patch release.

[asfimport]

Wes McKinney / @wesm:
In the meantime, I suggest we experiment with a build image starting from

https://github.com/pypa/manylinux/blob/master/docker/Dockerfile-x86_64#L1

but with centos6 instead of centos5. If there are users on ancient Linux, I actually think it would be good to hear from them when they have problems instead of assuming that they exist. I personally would be comfortable dropping support for glibc < 2.12 as long as that's the only thing we are breaking

[asfimport]

Wes McKinney / @wesm:
Seems like this probably merits a mailing list discussion. What do you think? At best we have a trade-off to make; my vote would be to choose progress and hope that pypa can catch up with the requirements of hybrid Python/C++ projects

[asfimport]

Antoine Pitrou / @pitrou:
Yes, we should poll the Arrow mailing-list(s) for this. Also, perhaps we should ask for guidance on distutils-sig, though I expect the answer there to be slightly negative :-)

[asfimport]

Wes McKinney / @wesm:
Done.

[asfimport]

Wes McKinney / @wesm:
It seems this is not pressing for 0.11 since we probably will not be shipping Abseil, so moving to 0.12

[asfimport]

Antoine Pitrou / @pitrou:
I also asked on distutils-sig:
https://mail.python.org/mm3/archives/list/[email protected]/thread/M4MSVY5MPAPXFWHH4PBLE6PEBPOBIA44/

[asfimport]

Antoine Pitrou / @pitrou:
Here is the meta-issue for manylinux2010 in Python packaging tools and infrastructure:
pypa/manylinux#179

At the minimum, I think we need to wait until auditwheel and pip support are merged in (perhaps also the Docker image?). Both have ready PRs but they seem to be stuck waiting for maintainer time.

[asfimport]

Wes McKinney / @wesm:
I still think it wouldn't be too evil to begin building with CentOS 6 as our base Linux for our wheels. gRPC is going to begin to use Abseil at some point from the look of the repo, and then we might be forced to do this anyway

[asfimport]

Wes McKinney / @wesm:
There is a docker image for manylinux2010 available now

https://discuss.python.org/t/manylinux2010-docker-image-now-available/1471

[asfimport]

Krisztian Szucs / @kszucs:
Issue resolved by pull request 4675
#4675