[Python 3.9 Upgrade] Docker CI Images update 3.7 to 3.9 (Part 1)

[peterzhuamazon]

Description

[Python 3.9 Update] Docker CI Images update 3.7 to 3.9 (Part 1).

There will be a part2 on the test ci images, this one is related to build ci images.
All the locales are removed to use the default ones:

CentOS7: C/Posix
Ubuntu2004: C/Posix
Rockylinux8: C.UTF-8

Thanks.

Issues Resolved

#3351

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #3634 (b23cd26) into main (bbf2bff) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #3634   +/-   ##
=======================================
  Coverage   91.48%   91.48%           
=======================================
  Files         181      181           
  Lines        5380     5381    +1     
=======================================
+ Hits         4922     4923    +1     
  Misses        458      458           

see 1 file with indirect coverage changes

[rishabh6788]

@peterzhuamazon Why don't we move to 3.10? Is it because it has too many breaking changes?

[peterzhuamazon]

@peterzhuamazon Why don't we move to 3.10? Is it because it has too many breaking changes?

Hi @rishabh6788. Python 3.9 has been tested on my local for 1+ months, and it is the default given version on al2023, and easily available on other OSes such as rockylinux8.

It is very stable for several years and it is much more similar to 3.7/3.8 than 3.10.

It is also only a year apart from 3.10 in terms of lts support. So considering all the circumstances it is the best option for us in terms of migration. Thanks.

[rishabh6788]

Approving assuming the docker images have already been built and tested thoroughly.

[peterzhuamazon]

Approving assuming the docker images have already been built and tested thoroughly.

The process has changes and will build after merge.
Thanks.

[peterzhuamazon]

Need to investigate a bit as the 3.9.17 from custom repo is having some issues when loading libs compares to the 3.9.5 version coming from default universe repo Ubuntu2004.

[peterzhuamazon]

Similar to this and missing python-apt as the re-link is messing up with default 3.8 version on Ubuntu2004: https://stackoverflow.com/questions/69949591/modulenotfounderror-no-module-named-apt-pkg-installing-deadsnakes-repository

[peterzhuamazon]

https://build.ci.opensearch.org/job/docker-build/3469/console
https://build.ci.opensearch.org/job/docker-build/3470/console
https://build.ci.opensearch.org/job/docker-build/3471/console
https://build.ci.opensearch.org/job/docker-build/3472/console
https://build.ci.opensearch.org/job/docker-build/3473/console
https://build.ci.opensearch.org/job/docker-build/3474/console

[peterzhuamazon]

https://build.ci.opensearch.org/job/docker-copy/537/console
https://build.ci.opensearch.org/job/docker-copy/538/console
https://build.ci.opensearch.org/job/docker-copy/539/console
https://build.ci.opensearch.org/job/docker-copy/540/console

[peterzhuamazon]

Seeing errors on the python compilation here on arm64:
https://build.ci.opensearch.org/job/docker-build/3474/console

#18 705.0 Exception in thread Thread-1:
#18 705.0 Traceback (most recent call last):
#18 705.0   File "/usr/local/lib/python3.9/threading.py", line 980, in _bootstrap_inner
#18 705.0     self.run()
#18 705.0   File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 323, in run
#18 705.0     self.terminate_broken(cause)
#18 705.0   File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 458, in terminate_broken
#18 705.0     work_item.future.set_exception(bpe)
#18 705.0   File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 549, in set_exception
#18 705.0     raise InvalidStateError('{}: {!r}'.format(self._state, self))
#18 705.0 concurrent.futures._base.InvalidStateError: CANCELLED: <Future at 0x5512854af0 state=cancelled>
#18 705.0 Listing '/usr/local/lib/python3.9/zoneinfo'...
#18 705.0 Traceback (most recent call last):
#18 705.0   File "/usr/local/lib/python3.9/compileall.py", line 459, in <module>
#18 705.0     exit_status = int(not main())
#18 705.0   File "/usr/local/lib/python3.9/compileall.py", line 436, in main
#18 705.0     if not compile_dir(dest, maxlevels, args.ddir,
#18 705.0   File "/usr/local/lib/python3.9/compileall.py", line 112, in compile_dir
#18 705.0     success = min(results, default=True)
#18 705.0   File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 562, in _chain_from_iterable_of_lists
#18 705.0     for element in iterable:
#18 705.0   File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
#18 705.0     yield fs.pop().result()
#18 705.0   File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 446, in result
#18 705.0     return self.__get_result()
#18 705.0   File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
#18 705.0     raise self._exception
#18 705.0 concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

[peterzhuamazon]

Not sure why this is the case, as other 2 centos7 images goes to success, tho this release image is a bit of outlier as it tries to compile openssl 1.1.1 as well as glibc higher versions.

[peterzhuamazon]

New try:
https://build.ci.opensearch.org/job/docker-build/3478/console
https://build.ci.opensearch.org/job/docker-build/3479/console

[peterzhuamazon]

https://build.ci.opensearch.org/job/docker-copy/541/console

[peterzhuamazon]

Seems like if there are two servers running build on the same time it will cause this error, and seems random, it will go away once we have rockylinux8 replace centos7.

[peterzhuamazon]

Try again: https://build.ci.opensearch.org/job/docker-build/3484/console

[peterzhuamazon]

Resolve python issues:

• [Python 3.9 Upgrade] Upgrade docker buildx 0.9.1 and fix Python compilation issues #3643