Add support for Python=3.12

[valeriupredoi]

Description

Closes #2227

Link to documentation: not yet available, to be posted

As of 28 May 2024

The prospector-related issue vs flake8 dependencies solved/workarounded in #2437 by @bouweandela (merged that branch here to have the full support)

Historical info

Only incompatibility is prospector - needs prospector-dev/prospector#645 that, in turn, needs a fix/workaround of master proposed in prospector-dev/prospector#658

Things are clear now: flake8==7.0.0 is not barfing at all - all clear (bar one actual PEP8 error that fixed in 57fcea4 ) - but, prospector is not happy to have latest flake8, am trying to push updated prospector pins in prospector-dev/prospector#645 - once that's merged, we will wait for a new prospector release, then I'll build a new prospector conda package myself. That should fix it all 👍

Incompatible packages

• none (as of 31 October 2023)

But flake8 is barfing! -> according to @zklaus @bouweandela and myself, the flake8 issue is a bug that I will report

EDIT on flake8 (as of 7 November 2023): thanks to tests done by @ehogan we now see flake8=6.1.0 almost not barfing at all (just a couple of errors, unlike flake8=5 that completely poops the bed, as seen from the GA tests linked); problem is, flake8=6.1.0 (latest version) is currently rejected by prospector; I'll have to go chase Carl at prospector get us a new version (probably with support for pylint >=3), then I'll rebuild it on feedstock.

Tests failing

• flake8 (see above) https://github.com/ESMValGroup/ESMValCore/actions/runs/6708774145/job/18230325803
• tests/unit/esgf/test_logon.py due to SafeConfigParser being removed completely from new Python (via esgf-pyclient via MyProxyClient); once replace the use of SafeConfigParser with bogstandard ConfigParser (and use direct import of configparser, not via six anymore) cedadev/MyProxyClient#20 is merged, and a new vesrion of MyProxyClient is on conda-forge, things will be a-ok (I tested with the dev branch in that PR) This has now been fixed with a new version of MyProxyClient and a rebuild of esgf-pyclient for Python 3.12 (both by yours truly)

Historical incompatible packages (resolved)

• cartopy (incompatibility tested on: 11 October 2023)
• cf-units (incompatibility tested on: 11 October 2023)
• iris via cf-units
• iris-esmf-regrid via cf-units
• fiona via gdal/pillow. even if fiona=1.9.4 has a py312 build (incompatibility tested on: 11 October 2023)

These seem to be the only ones - env solves fine with these out of the picture! Fiona plan on a 1.9.5 in a day or two anyway Toblerity/Fiona#1301 - gonna take up the Python=3.12 with the MO folk then 👍

Work started elsewhere

• Bump build number for rerendering conda-forge/cf_units-feedstock#56

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🛠 This pull request has a descriptive title and labels
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 94.51%. Comparing base (515f399) to head (80325ce).
Report is 39 commits behind head on main.

Files	Patch %	Lines
esmvalcore/_task.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2228   +/-   ##
=======================================
  Coverage   94.51%   94.51%           
=======================================
  Files         246      246           
  Lines       14024    14024           
=======================================
  Hits        13255    13255           
  Misses        769      769           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zklaus]

The Flake8 errors seem to be a Flake8 bug, where that tool checks f-strings and complains about missing space after : for example in f'http://something.org/ABC/v1/{filename}'.

[valeriupredoi]

indeed @zklaus - next hurdle A mistake in the recipe https://github.com/ESMValGroup/ESMValCore/actions/runs/6719725575/job/18261890333?pr=2228 - I do wonder who wrote that terse error message 🤣

[valeriupredoi]

indeed @zklaus - next hurdle A mistake in the recipe https://github.com/ESMValGroup/ESMValCore/actions/runs/6719725575/job/18261890333?pr=2228 - I do wonder who wrote that terse error message 🤣

fixed! Now on to the next hurdle...

[valeriupredoi]

the myproxy import error masks the real problem:

raceback (most recent call last):
  File "/home/valeriu/ESMValCore/testimp.py", line 1, in <module>
    from myproxy.client import MyProxyClient
  File "/home/valeriu/miniconda3/envs/esmvaltool-fromlock/lib/python3.12/site-packages/myproxy/client/__init__.py", line 44, in <module>
    from myproxy.client.utils.openssl import OpenSSLConfig
  File "/home/valeriu/miniconda3/envs/esmvaltool-fromlock/lib/python3.12/site-packages/myproxy/client/utils/__init__.py", line 11, in <module>
    from six.moves.configparser import SafeConfigParser
ImportError: cannot import name 'SafeConfigParser' from 'configparser' (/home/valeriu/miniconda3/envs/esmvaltool-fromlock/lib/python3.12/configparser.py). Did you mean: 'RawConfigParser'?

[valeriupredoi]

OK SafeConfigParser -> ConfigParser and complete retirement in Python 3.12: philipkershaw/MyProxyClient#9 - correction, since repo moved, issue is here cedadev/MyProxyClient#19 - will followup with a PR

[valeriupredoi]

the PR is there cedadev/MyProxyClient#20 - I have spoken with @agstephens in the meantime, and he was kind enough to see if we can get that through @alaniwi and merged, so we can up the version or build of MyProxyClient. This is an unforlding situation, like BBC says 😁

[valeriupredoi]

@bouweandela prospector pin should be excluded of setup.py since it conflicts with the already installed flake8 see https://github.com/ESMValGroup/ESMValCore/actions/runs/9268331024/job/25496728780?pr=2228

[valeriupredoi]

OK I'll revoke that too - prospector needs altogether removed from setup.py since the development installation grabs a very very old version on top of the already existing conda-forge one https://github.com/ESMValGroup/ESMValCore/actions/runs/9268407947/job/25496971870?pr=2228

[valeriupredoi]

OK final env: prospector==1.10.3 from conda-forge, flake8==7 from pypi 🥳 https://github.com/ESMValGroup/ESMValCore/actions/runs/9268654163/job/25497753109?pr=2228

[bouweandela]

@valeriupredoi Do you want to run the actions again? Or do you think it's ok?

[bouweandela]

@chrisbillowsMO We would like to include these changes in the upcoming release if possible. We can still keep the pin on Python <3.12 for the ESMValTool v2.11 package, but this will allow us to start switch the development version of ESMValTool v2.12 with Python 3.12 once all dependencies are up to date.

[valeriupredoi]

@bouweandela thanks a lot, bud! Only took us 8 months 🤣

[chrisbillowsMO]

@chrisbillowsMO We would like to include these changes in the upcoming release if possible. We can still keep the pin on Python <3.12 for the ESMValTool v2.11 package, but this will allow us to start switch the development version of ESMValTool v2.12 with Python 3.12 once all dependencies are up to date.

Hi @bouweandela - have added to our checklist here: #2389 (comment)

[valeriupredoi]

many thanks @chrisbillowsMO 🍺