Migrate to Python 3.12, require it

[PallHaraldsson]

I realize you support older, and may want to, I'm just thinking if it could be better to only support 3.12+ already.

What might be reasons and pros and cons? I see:

Porting to Python 3.12

• Legacy Unicode APIs based on Py_UNICODE* representation has been removed. Please migrate to APIs based on UTF-8 or wchar_t*.

They want to move to UTF-8, I believe it's not yet the default encoding, but it's been an alternative redundant encoding for a long time.

When you call to or from Python, then I believe it's non-copying, you can reuse arrays in either direction, and that would be ideal for strings too and all else. What are the exceptions? Strings are UTF-8 in Julia and need to be converted (or not?), if calling Python, though probably if Julia is called with a Python string.

Also Dicts are unordered in Julia, so I'm not sure what happens. Possibly OrderedDefaultDict could be handled better, or if were Julia's default, but right now dicts can't be shared in either direction. Are there more exceptions?

Python 3.12 should work already, so you don't have to do anything per se. And PythonCall will install a Python for you (now a smaller download since a lot of deprecated/outdated stuff dropped), but if you require 3.12 or higher it will be 3.12 for now. I think you already get the latest anyway? Unless you ask for older directly or indirectly, i.e. some package forces it. So then it couldn't be used. Going forward will not be a problem.

The only point of restricting is if you add capability that only exists in 3.12, so that you can use it. Older versions of your package would still work with older Python and that could be documented, and even supported for a while.

So what do you think, is there anything new and interesting in 3.12 you want to take advantage of?

I haven't looked over everything, but there are intriguing changes (probably largest Python release in a while):

[Python already has a (limited) stable C API, that I suppose you use. Plus I now see this new unstable.]

New Features¶

• PEP 697: Introduce the Unstable C API tier, intended for low-level tools like debuggers and JIT compilers. This API may change in each minor release of CPython without deprecation warnings. Its contents are marked by the PyUnstable_ prefix in names.

[..]

• PEP 697: Add an API for extending types whose instance memory layout is opaque:
  Add the new limited C API function PyType_FromMetaclass(), which generalizes the existing PyType_FromModuleAndSpec() using an additional metaclass argument. (Contributed by Wenzel Jakob in gh-93012.)

• API for creating objects that can be called using the vectorcall protocol was added to the Limited API:
  [..]
  The Py_TPFLAGS_HAVE_VECTORCALL flag is now removed from a class when the class’s __call__() method is reassigned. This makes vectorcall safe to use with mutable types (i.e. heap types without the immutable flag, Py_TPFLAGS_IMMUTABLETYPE). Mutable types that do not override tp_call now inherit the Py_TPFLAGS_HAVE_VECTORCALL flag. (Contributed by Petr Viktorin in gh-93274.)
  The Py_TPFLAGS_MANAGED_DICT and Py_TPFLAGS_MANAGED_WEAKREF flags have been added. This allows extensions classes to support object __dict__ and weakrefs with less bookkeeping, using less memory and with faster access.

[..]

• The C API now permits registering callbacks via PyDict_AddWatcher(), PyDict_Watch() and related APIs to be called whenever a dictionary is modified. This is intended for use by optimizing interpreters, JIT compilers, or debuggers. (Contributed by Carl Meyer in gh-91052.)

[..]

• PEP 683: Introduce Immortal Objects, which allows objects to bypass reference counts, and related changes to the C-API:

[..]

• PEP 684: Add the new Py_NewInterpreterFromConfig() function and PyInterpreterConfig, which may be used to create sub-interpreters with their own GILs. (See PEP 684: A Per-Interpreter GIL for more info.) (Contributed by Eric Snow in gh-104110.)

• In the limited C API version 3.12, Py_INCREF() and Py_DECREF() functions are now implemented as opaque function calls to hide implementation details. (Contributed by Victor Stinner in gh-105387.)

[MilesCranmer]

Just passing by to comment that as a maintainer of a Python-Julia downstream package (PySR), I would perhaps vote against this change, since some of my users have to use their institute cluster Python which is typically fixed to an older version.

I think it's important for tools at the upper levels of a dependency hierarchy, like PythonCall.jl, to impose as few subjective constraints as possible, as they have a cascading effect.

My suggestion would be to simply follow the "Security support" column of https://endoflife.date/python which is 3.8-3.12 currently. Older versions are not receiving security updates. 3.11 is still officially supported with security updates by the Python Software Foundation until October 24, 2027 – which would make that a good point at which to switch to 3.12 only. So my suggestion is to table this until then. Wdyt?

[PallHaraldsson]

Older versions of Python would still work in older PythonCall.jl at least. I'm thinking for the future, to be able to rely on new features in Python, at least UTF-8 if/when it will be default. If there's nothing compelling to know we can depend on then it's a moot point.

Also I was thinking default/require 3.12, but possibly still in latest PythonCall.jl there could be a workaround to still allow older, or automatically support and then with reduced (i.e. current) functionality.

About that institute cluster Python, since it's older version than 3.12, would it not help if PythonCall can (and already does) download latest for you?

[MilesCranmer]

Maybe one could just wrap new features with a

import sys
if sys.version_info >= (3, 12, 0):
    ...

block? I know some Python packages that do stuff like this. I guess if you want to make structural changes then it becomes a bit bloated though, so not sure.

Re: cluster python, I agree that my personal preference is to just have my own python version. But I am aware of cases where sysadmin can get annoyed by this because it can blow up the file count on $HOME (e.g., some conda environments can have 100,000 files!), which itself slows down backups, compared to if it is a centrally managed python with common packages shared among all users. So some users are simply stuck with a particular python version and need to link it to any software they use. (This is obviously institute-dependent though!)

[PallHaraldsson]

This intriguing UTF-8 bug on Windows was fixed, and backported, but only back to 1.10:
https://bugs.python.org/issue37871

I didn't confirm it could be triggered by Julia, but very plausibly, so at least recommend 1.10 or later?

[cjdoris]

Maybe this isn't written down anywhere and should be but I'm aiming to remain compatible with all supported versions of Julia and Python unless there are highly compelling reasons not to.

I'm not aware of any sufficiently compelling functionality in 3.12 (including what's pointed out above) so I'll close this issue.