Prevent abi3 from being used with no-GIL interpreter

[abravalheri]

Summary of changes

Closes #4420

Pull Request Checklist

• Changes have tests
• News fragment added in newsfragments/.
  (See documentation for details)

[abravalheri]

@dholth, @bastimeyer, this is a draft that expands on #4420 (comment) (I just wanted to make sure the information why abi3 is not used is present in the logs).

Does it make sense to still use the given py_limited_api and just ignore abi3, or we should also discard the given py_limited_api? (If the second option is preferable, then it would make more sense to me to raise an exception due to incompatible configuration).

I don't have an approach to test this in the CI yet... Suggestions are welcome.

[bastimeyer]

As explained in the issue thread, impl_ver does not include abi flags. Here's the implementation of tags.interpreter_version():

setuptools/setuptools/_vendor/packaging/tags.py

Lines 535 to 544 in f91fa3d

	def interpreter_version(*, warn: bool = False) -> str:
	"""
	Returns the version of the running interpreter.
	"""
	version = _get_config_var("py_version_nodot", warn=warn)
	if version:
	version = str(version)
	else:
	version = _version_nodot(sys.version_info[:2])
	return version

>>> import sys,sysconfig
>>> sysconfig.get_config_var("py_version_nodot")
'313'
>>> sys.abiflags
't'
>>> sys.version_info[:2]
(3, 13)

[abravalheri]

Thanks @bastimeyer.

It seems that we can use sys.abiflags then...

[abravalheri]

I have updated the code to reflect my latest understanding.

[abravalheri]

I don't know how to deal with the following problem however:

The extensions themselves define py_limited_api like in https://github.com/Legrandin/pycryptodome/blob/39b5a627dab9124da0fb9d15a2f3a19434d96565/setup.py#L286C9-L286C28.

Not sure how that will interact with this.

Maybe the best is to just raise an exception and require the developer to be mindful about the compatibility with py_limited_api (and change their setup.py accordingly to react to the current version of Python being used)?

[dholth]

Does that really break, or is a "py_limited_api" extension, compiled for the nogil intepreter, just not cross-compatible?

[abravalheri]

This is a bit out of my comfort zone for the time being, but my understanding of the discussion on python/cpython#111506, https://discuss.python.org/t/python-abis-and-pep-703/34018 is that there is potential for break things.

The following hints in https://peps.python.org/pep-0703/#backwards-compatibility seem to suggest to me that some parts of the limited API may currently require (or at least assume) the GIL to work properly (therefore extensions using those parts of the limited API are potentially prone to runtime errors).

• C-API extensions that rely on the GIL to protect global state or object state in C code will need additional explicit locking to remain thread-safe when run without the GIL.
• C-API extensions that use borrowed references in ways that are not safe without the GIL will need to use the equivalent new APIs that return non-borrowed references. Note that only some uses of borrowed references are a concern; only references to objects that might be freed by other threads pose an issue.
• Custom memory allocators (PyMem_SetAllocator) are required to delegate the actual allocation to the previously set allocator. For example, the Python debug allocator and tracing allocators will continue to work because they delegate the allocation to the underlying allocator. On the other hand, wholesale replacing of the allocator (e.g., with jemalloc or tcmalloc) will not work correctly.
• Python objects must be allocated through the standard APIs, such as PyType_GenericNew or PyObject_Malloc. Non-Python objects must not be allocated through those APIs. For example, it is currently acceptable to allocate buffers(non-Python objects) through PyObject_Malloc; that will no longer be allowed and buffers should instead be allocated through PyMem_Malloc, PyMem_RawMalloc, or malloc.

[bastimeyer]

abravalheri requested a review from bastimeyer last week

#4424 (comment)

I don't know how to deal with the following problem however:

The extensions themselves define py_limited_api like in Legrandin/pycryptodome@39b5a62/setup.py#L286C9-L286C28

I'm sorry, but I can't help with this, because as explained in the issue thread, I don't have any experience with Python C-extensions, so I don't know the specific requirements and the best way forward in order to solve this. The issue was opened by me purely from the perspective of someone who has a dependency on a project with C-extensions and abi3 wheels, where I ran into these build issues.

The maintainers of those projects which build extensions based on the limited API will need to provide feedback. I've already asked the maintainers of pycryptodome in the issue I've opened there, but I can ask again and link to this pull request, too.

[rgommers]

Automatically disabling building for the limited API or raising a clear exception telling the user to set a flag to explicitly disable use of the limited API are both reasonable options I believe. As a data point: meson-python does the latter (see mesonbuild/meson-python#621 and docs) while scikit-build-core does the former (see scikit-build/scikit-build-core#741 and docs).

[abravalheri]

Based on the feedback in #4424 (comment), I opted to raise a clean exception instead of the warning.

The message the user will see is something like:

`py_limited_api='cp35'` not supported. `Py_LIMITED_API` is currently incompatible with
`Py_GIL_DISABLED` (sys.abiflags='t'). See https://github.com/python/cpython/issues/111506.

which make it easier to identify the root cause of the problem and is better than what they currently see:

AssertionError: would build wheel with unsupported tag ('cp35', 'abi3', 'linux_x86_64')

ERROR Backend subprocess exited when trying to invoke build_wheel

[abravalheri]

@jaraco do you agree with this approach?

[jaraco]

Sounds good to me. I don't have more than a superficial understanding of the concern. For example, it's unclear to me if this limitation is temporary and if so, under what conditions could the restriction be lifted. I hope in the future some people will bring more insight and help provide more robust support for the various combinations (if feasible).