Add manylinux_2_24 images

[mayeut]

This adds manylinux_2_24 images and rework the master branch to build manylinux2010, manylinux2014 and manylinux_2_24 from a single branch.

The build scripts were reworked for maintainers to be able to rebuild images taking advantage of docker cache.
This obviously has pros and cons compared to the previous one step build scripts. Here are my thoughts:
pros:

• docker cache (i.e. faster build time for local builds)
• one build script file per tool, I think this makes things clearer and allows to customize tool depending on the image being built without having a single script with many if
• homogeneous tools available across images
• image size is kept to a minimum (at the moment - and to my knowledge - yum or apt are making our life difficult when it comes to removing packages to restore a previous state, i.e. something like alpine's apk add --virtual build-deps .../apk del build-deps)

cons:

• dependency versions are scattered across different files
• docker handling of intermediate build stage images could be better

Please do provide feedback about rewriting as proposed. If you have other proposals on how you think it should be done, please do tell.

Some of the scripts might not be considered docker cache friendly. This is especially the case for install-entrypoint.sh, install-runtime-packages.sh and install-build-packages.sh. My motivation for doing things like this is that it helps getting consistency between images and those files are likely to change only when adding a new set of images which will not happen that often I guess. For install-build-packages.sh, it might also change when adding a new tool. This will have an impact on all images so should not be an issue.

This removes CPython 2.7 on manylinux2010 image.

README is not updated yet, waiting for feedback on other parts before doing this.

PS: This PR is also merging manylinux2014 in master not to lose history. This accounts for the high number of commits. I can open another PR targeting manylinux2014 branch first if this helps review.

[mattip]

How would you suggest we proceed with getting this reviewed and merged? It is a massive PR. Is there a way to break it into pieces either as separate PRs or at least some kind of "guide for reviewers"?

[mattip]

Or maybe set up a synchronous call with some PyPA maintainers to walk them through it?

[mayeut]

@mattip,

As mentioned in "PS":

This PR is also merging manylinux2014 in master not to lose history. This accounts for the high number of commits. I can open another PR targeting manylinux2014 branch first if this helps review.

I created #873 to at least allow a review commit by commit. Nevertheless, commit 76dee71 will be a big one (breaks the "almost monolithic" build.sh/build_utils.sh in 1 script per tool meaning everything that was removed from those are to be found in the new files). I'm not sure how I can break this in a more "reviewable" way but I'll try to think of something over the week-end.

pinging @pypa/manylinux if they want to chime in on the way to proceed forward.

[mayeut]

I will open one PR per tool migration for 76dee71 meaning I'll also do a few PR before that that build one on top of the others. So be it for the docker cache on my machine. I'll let travis-ci do the check and only build locally on errors if any in the meantime.

I can't get this PR back in draft, do not merge this.

[trishankatdatadog]

I can't get this PR back in draft, do not merge this.

I can apparently mark it as draft. Would you like me to?

[mayeut]

I propose the following strategy in order to get "this" merged:
Open PRs that will build one on top of the other requiring them to be reviewed sequentially which might take some time overall but will really ease the reviewing process with smaller changes. I won't open them all at once as they will depend on one another and reviews in a previous PR might lead to too much rework in the following PRs but I will open a couple at a time and rebase as PR gets merged in.
The first steps will be to merge manylinux2010 and manylinux2014 builds in the master branch. Once this is done, I can propose an extension for manylinux_2_24 images which will only have small changes compared to the new master.

I will create an issue to track the progress (but I'll keep this PR opened for now as a reference to where I'd like things to converge).

Edit: issue #877 created to discuss this further

[trishankatdatadog]

You might find Projects helpful to track this work.

[h-vetinari]

Seems this needs a rebase after #993 and all the work from #877 (GH still shows the commits from the manylinux2014 branch).

[mayeut]

Superseded by #999