gh-104469: Disallow using Py_LIMITED_API with Py_BUILD_CORE

[vstinner]

• Issue: Improve the readability and maintainability of test_capi using the AC #104469

[vstinner]

When I reviewed #107857 I noticed that Py_LIMITED_API was used with Py_BUILD_CORE which makes no sense to me. Make a choice: either the limited C API, or the internal C API :-)

[vstinner]

@corona10 wrote:

Okay, this is an issue between AC and limited C APIs. We should revert it first.
cc @erlend-aasland @vstinner @nahyeon-an

In the meanwhile, AC got support for the limited C API ;-)

[vstinner]

Merged. Thanks for the review @colesbury.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL7 LTO + PGO 3.x has failed when building commit 3f5c564.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/244/builds/5490) and take a look at the build logs.
4. Check if the failure is related to this commit (3f5c564) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/244/builds/5490

Failed tests:

• test.test_concurrent_futures.test_shutdown

Failed subtests:

• test_interpreter_shutdown - test.test_concurrent_futures.test_shutdown.ProcessPoolSpawnProcessPoolShutdownTest.test_interpreter_shutdown

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/test/test_concurrent_futures/test_shutdown.py", line 49, in test_interpreter_shutdown
    self.assertFalse(err)
AssertionError: b'Exception in thread Thread-1:\nTraceback (most recent call last):\n  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/threading.py", line 1059, in _bootstrap_inner\n    self.run()\n  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/concurrent/futures/process.py", line 339, in run\n    self.add_call_item_to_queue()\n  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/concurrent/futures/process.py", line 394, in add_call_item_to_queue\n    self.call_queue.put(_CallItem(work_id,\n  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/multiprocessing/queues.py", line 94, in put\n    self._start_thread()\n  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/multiprocessing/queues.py", line 177, in _start_thread\n    self._thread.start()\n  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/threading.py", line 978, in start\n    _start_new_thread(self._bootstrap, ())\nRuntimeError: can\'t create new thread at interpreter shutdown\nTraceback (most recent call last):\n  File "<string>", line 1, in <module>\n  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/multiprocessing/spawn.py", line 122, in spawn_main\n    exitcode = _main(fd, parent_sentinel)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/multiprocessing/spawn.py", line 132, in _main\n    self = reduction.pickle.load(from_parent)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build/Lib/multiprocessing/synchronize.py", line 115, in __setstate__\n    self._semlock = _multiprocessing.SemLock._rebuild(*state)\n                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n' is not false

[erlend-aasland]

Thanks! LGTM

[vstinner]

Oh. This change broke WASM buildbots.

Example with wasm32-emscripten node (dynamic linking) 3.x: https://buildbot.python.org/all/#/builders/1056/builds/3142

/opt/emsdk/upstream/emscripten/emcc  -DNDEBUG -g -O3 (...) -DPy_BUILD_CORE_BUILTIN -c ../../Modules/_testcapi/vectorcall_limited.c -o Modules/_testcapi/vectorcall_limited.o

In file included from ../../Modules/_testcapi/vectorcall_limited.c:2:
In file included from ../../Modules/_testcapi/parts.h:7:
In file included from ../../Include/Python.h:44:
../../Include/pyport.h:52:4: error: "Py_LIMITED_API is not compatible with Py_BUILD_CORE"
#  error "Py_LIMITED_API is not compatible with Py_BUILD_CORE"
   ^
1 error generated.

I'm not sure how -DPy_BUILD_CORE_BUILTIN landed in the command building Modules/_testcapi/vectorcall_limited.c.

I don't think that it's correct that vectorcall_limited.c which tests the limited C API on purpose it built with Py_BUILD_CORE.

On my Linux machine, Makefile contains:

Modules/_testcapi/vectorcall_limited.o: $(srcdir)/Modules/_testcapi/vectorcall_limited.c $(MODULE__TESTCAPI_DEPS) $(MODULE_DEPS_SHARED) $(PYTHON_HEADERS); $(CC) $(MODULE__TESTCAPI_CFLAGS) $(PY_STDMODULE_CFLAGS) $(CCSHARED) -c $(srcdir)/Modules/_testcapi/vectorcall_limited.c -o Modules/_testcapi/vectorcall_limited.o

So it gets two groups of compiler flags:

• MODULE__TESTCAPI_CFLAGS: not defined (empty)
• PY_STDMODULE_CFLAGS: -fno-strict-overflow -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -Og -Wall -O0 -std=c11 -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include

I don't see -DPy_BUILD_CORE_BUILTIN here.

[vstinner]

Oh. This change broke WASM buildbots.

I created issue #109723 to track this issue.