gh-99113: A Per-Interpreter GIL!

[ericsnowcurrently]

This is the culmination of PEP 684 (and of my 8-year long multi-core Python project)!

Each subinterpreter may now be created with its own GIL (via Py_NewInterpreterFromConfig()). If not so configured then the interpreter will share with the main interpreter--the status quo since the subinterpreters were added decades ago. The main interpreter always has its own GIL and subinterpreters from Py_NewInterpreter() will always share with the main interpreter.

(FYI, I've split up the original gh-99114 into multiple PRs, with this one being the final one in the chain.)

• Issue: Implement PEP 684: A Per-Interpreter GIL #99113

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit 289d050 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[ericsnowcurrently]

FYI, there are plenty of little things to clean up after this lands, particularly for PEP 684. There are also some corner-case crashes I need to chase down, related to extreme races.

Also, I still can't believe I've finally reached this point with this project!!! 😭

[erlend-aasland]

LGTM 🎉 BTW, I think the massive top-level comment in Python/ceval_gil.c needs an overhaul now.

[kumaraditya303]

LGTM is short for the amount of effort you put into it.

[ericsnowcurrently]

Thanks for all the great work and support, everyone!

There are plenty of small things to wrap up, but this is definitely the climax.

[aotto1968]

well , i changed my library to use this, but end up in ...

I started using Py_NewInterpreterFromConfig to add an PER-THREAD interpreter with my
library. The library is thread safe and supports many languages
-> C C++ JAVA C# VB.Net Perl Python GO TCL Ruby Php
other languages with thread support
-> C C++ JAVA C# TCL GO
works fine.
even python works fine without thread support.

Task: now I started thread support in python, I had to change to Multi-phase initialization
isolate the python data on an per-thread-level and finally gent an easy thread example working
-> ok

    static PyInterpreterConfig config = {
        .use_main_obmalloc = 0,
        .allow_fork = 0,
        .allow_exec = 0,
        .allow_threads = 1,
        .allow_daemon_threads = 0,
        .check_multi_interp_extensions = 1,
        .gil = PyInterpreterConfig_OWN_GIL,
    };

    PyThreadState *tstate = NULL;
    PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);

NOW I start with an more complex example and this stuff fails with a python INTERNAL core
that I can't debug. The stack-trace comes from my MqDisasterSignal handler.

-> question: what can I do to debug this issue?
-> debugging with valgrind is not possible because with valgrind this example works fine.


Debug memory block at address p=0x7f11884def70: API '^@'
    18302063728033398269 bytes originally requested
    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
        at p-7: 0x00 *** OUCH
        at p-6: 0x00 *** OUCH
        at p-5: 0x00 *** OUCH
        at p-4: 0x00 *** OUCH
        at p-3: 0x00 *** OUCH
        at p-2: 0x00 *** OUCH
        at p-1: 0x00 *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 8 pad bytes at tail=0xfdfe7d0f864bed6d are X> {PRINT               :pid(60831):tid(0x7f118819c700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1cb99b8     ):MqDisasterSignal              }: BackTrace {
:pid(60831):tid(0x7f118819c700):   [ library              : filename                       : lineno ] function
:pid(60831):tid(0x7f118819c700):   [ -------              : --------                       : ------ ] --------
:pid(60831):tid(0x7f118819c700):   [ theLink              : c/sys_mq.c                     : 705    ] MqDisasterSignal
:pid(60831):tid(0x7f118819c700):   [ unknown              : unknown                        : 0      ] unknown
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2408   ] _PyObject_DebugDumpAddress
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2326   ] _PyMem_DebugCheckAddress
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2159   ] _PyMem_DebugRawFree
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 685    ] PyMem_RawFree
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 1853   ] _PyObject_Free
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2163   ] _PyMem_DebugRawFree
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 2296   ] _PyMem_DebugFree
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/obmalloc.c             : 830    ] PyObject_Free
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1569   ] dictresize
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1194   ] insertion_resize
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1261   ] insertdict
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1865   ] _PyDict_SetItem_Take2
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/dictobject.c           : 1883   ] PyDict_SetItem
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7618   ] add_subclass
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7352   ] type_ready_add_subclasses
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7515   ] type_ready
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 7553   ] PyType_Ready
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 3795   ] type_new_impl
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 3929   ] type_new
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 1664   ] type_call
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 240    ] _PyObject_MakeTpCall
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/typeobject.c           : 3949   ] type_vectorcall
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 133    ] _PyObject_FastCallDictTstate
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 157    ] PyObject_VectorcallDict
:pid(60831):tid(0x7f118819c700):   [ system               : Python/bltinmodule.c           : 208    ] builtin___build_class__
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/methodobject.c         : 438    ] cfunction_vectorcall_FASTCALL_KEYWORDS
:pid(60831):tid(0x7f118819c700):   [ system               : Include/internal/pycore_call.h : 92     ] _PyObject_VectorcallTstate
:pid(60831):tid(0x7f118819c700):   [ system               : Objects/call.c                 : 325    ] PyObject_Vectorcall
:pid(60831):tid(0x7f118819c700):   [ system               : Python/bytecodes.c             : 2714   ] _PyEval_EvalFrameDefault
:pid(60831):tid(0x7f118819c700):   [ system               : nclude/internal/pycore_ceval.h : 89     ] _PyEval_EvalFrame
:pid(60831):tid(0x7f118819c700):   [ system               : Python/ceval.c                 : 1683   ] _PyEval_Vector
:pid(60831):tid(0x7f118819c700):   [ system               : Python/ceval.c                 : 578    ] PyEval_EvalCode
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 1722   ] run_eval_code_obj
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 1743   ] run_mod
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 1643   ] pyrun_file
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 433    ] _PyRun_SimpleFileObject
:pid(60831):tid(0x7f118819c700):   [ system               : Python/pythonrun.c             : 466    ] PyRun_SimpleFileExFlags
:pid(60831):tid(0x7f118819c700):   [ theLink              : py/MqFactoryC_py.c             : 194    ] py_mqmsgque_sFactoryCTor
:pid(60831):tid(0x7f118819c700):   [ theLink              : c/MqFactoryS_mq.c              : 289    ] MqFactoryInvoke_RT
:pid(60831):tid(0x7f118819c700):   [ theLink              : c/sys_mq.c                     : 269    ] MqSysServerThreadMain
:pid(60831):tid(0x7f118819c700):   [ theLink              : c/sys_mq.c                     : 324    ] sSysServerThreadInit
:pid(60831):tid(0x7f118819c700):   [ unknown              : unknown                        : 0      ] unknown
:pid(60831):tid(0x7f118819c700): }
:pid(60831):tid(0x7f118819c700): ^[[1;36mglobal data -----------------------------------------------------^[[0;m
:pid(60831):tid(0x7f118819c700): ^[[1;36mruntime  : MkRuntimeRLS=0x1cb93b0, MqRuntimeRLS=0x7f11879539c8^[[0;m
:pid(60831):tid(0x7f118819c700): ^[[1;36mgeDat    : geDat=0x7f1188499de0, geMax=16, geCur=2^[[0;m
:pid(60831):tid(0x7f118819c700): ^[[1;36mevent    : event=0x7f1187954610, DataLNum=16, DataLCur=0, geNum=1, fdmax=0^[[0;m
.../NHI1/bin/Nhi1Exec: line 870: 60831 Aborted                 '.../x86_64-suse-linux-gnu/debug/bin/python3' '.../NHI1/example/py/server.py' '--thread' '--uds' '--file' '/tmp/test.uds'