Redesign scripts to build all images from a single branch

[mayeut]

As proposed in #542 (comment), this issue is created to propose and track progress of a rewrite of manylinux build scripts to build all images from a single branch.

With new manylinux policies allowed by PEP600, we'll have more and more images. Creating a branch per policy will cause issues for maintainability and consistency between images (or even README).

manylinux1 is out-of-scope for the rewrite for now (c.f. #542 (comment) for rational)

Here are the steps I propose, feel free to challenge those.
PRs referenced here will build one on top of the other requiring them to be reviewed sequentially.

• Rework the manylinux2014 branch to allow for multiple policies
  Rational for choosing this as the base branch of the rework: manylinux2014 branch builds on all platforms which allows to see issues that would otherwise not be seen until a later point.

  • Use graviton2 for arm64 build ([manylinux2014] Switch arm64 to using Travis-CI Graviton 2 #875)
  • Use a single Dockerfile for all images ([manylinux2014] Use a single Dockerfile for all images #876)
  • Rework scripts to be docker cache friendly (mostly for maintainers doing local builds)
    • Round 1 ([manylinux2014] Rework scripts to be more docker cache friendly #879)
    • Round 2 ([manylinux2014] Split libxcrypt build in runtime_base image #965)
    • Round 3 ([manylinux2014] Split patchelf build in runtime_base image #966)
    • Round 4 ([manylinux2014] Split libtool build in runtime_base image #967)
    • Round 5 ([manylinux2014] Split automake build in runtime_base image #968)
    • Round 6 ([manylinux2014] Split cmake build from build_base image #969)
    • Round 7 ([manylinux2014] Split git build from build_base image #970)
    • Round 8 ([manylinux2014] Split cpython build from build_base image #971)
    • Round 9 ([manylinux2014] Split sqlite3 build in build_cpython image #972)
    • Round 10 ([manylinux2014] Split cpython build by minor version #973)
    • Round 11 ([manylinux2014] Hardlink /opt/_internal to save 60MB #974)
    • Round 12 ([manylinux2014] Strip tools installed in runtime_base image #975)
    • Round 13 ([manylinux2014] Add swig #976)
    • Round 14 ([manylinux2014] Update autoconf from 2.69 to 2.70 #977)
    • Round 15 ([manylinux2014] Update automake from 1.16.2 to 1.16.3 #978)

• Merge manylinux2014 in master
  At this point, we'll be ready to add new images for newer PEP600 policies

  • Update glibc on manylinux2010 ([manylinux2010] glibc update & glibc cache/image size reduction #874)
  • Publish quay.io/pypa/manylinux2010_centos-6-no-vsyscall (Publish quay.io/pypa/manylinux2010_centos-6-no-vsyscall #878)
  • Add manylinux2010 build support in manylinux2014 branch ([manylinux2014] Allow manylinux2010 build from manylinux2014 branch #979)
  • Merge in master (Merge manylinux2014 branch in master branch #993)

• Add manylinux_2_24 images (Add manylinux_2_24 images #999)
  Rational for adding this specific policy: It solves issues for numpy ppc64le c.f. Tracking issue for PEP 600 Perennial manylinux rollout #542 (comment), numpy being the most downloaded manylinux package it seems a good starting point as it's also probably used as a dependency in a high number of manylinux packages which will thus benefit from this.

@pypa/manylinux, please do provide feedback as your time allows, thanks.

[manthey]

With some of these changes, the docker image now has many more layers. Docker has an upper limit on the total number of layers that an image can have (it is around 128, but I am not certain), and having more layers here mean that downstream docker images that build on it have fewer layers that they can add. I recognize that having more layers makes these scripts more docker cache friendly for manylinux, but more docker cache hostile for downstream docker images. Please don't add too many more layers.

[mayeut]

@manthey,
I think the total number of layers is 127. This added 31 layers compared to the previous method (now 49 layers, previously 18 layers if I'm not mistaken). If this becomes an issue, I have ideas in order to reduce the number of layers.
We could easily get 5 layers back at no cost. Saving more layers would "cost" storage for cache.
As long as it's not an issue, I'll let things as they are. Thanks for pointing out this potential issue.

[manthey]

Thanks, @mayeut.