Crash when using sqlite3 with subinterpreters

[vstinner]

Running sqlite3_crash.py on Python 3.9 and 3.10 fail randomly with an assertion failure in _PyObject_GC_UNTRACK() called by func_dealloc().

The problem is that a function is allocated in an interpreter and deallocated in another interpreter.

I built Python 3.10 with pyobject_ob_interp.patch. When the bug occurs, I get:

• func_dealloc(): ob_interp = 0x7fffa4001030
• In a parent frame (_PyObject_VectorcallTstate), tstate->interp = 0x7fffc0001030

[vstinner]

In the main branch, the _sqlite3 extension was converted to the multi-phase initialization API (PEP 489) which made the extension safer to use with subinterpreters.

[vstinner]

See also #90228 which is similar but has been fixed.

[vstinner]

In the 3.10 branch, the script doesn't crash before my change:

commit e6bb17fe29713368e1fd93d9ac9611017c4f570c (HEAD)
Author: Miss Islington (bot) <[email protected]>
Date:   Thu Jan 13 10:50:09 2022 -0800

    bpo-46070: _PyGC_Fini() untracks objects (GH-30577)
    
    
    Py_EndInterpreter() now explicitly untracks all objects currently
    tracked by the GC. Previously, if an object was used later by another
    interpreter, calling PyObject_GC_UnTrack() on the object crashed if
    the previous or the next object of the PyGC_Head structure became a
    dangling pointer.
    (cherry picked from commit 1a4d1c1c9b08e75e88aeac90901920938f649832)
    
    Co-authored-by: Victor Stinner <[email protected]>

[erlend-aasland]

I can reproduce this on my Mac on 3.9 and 3.10.

[vstinner]

It seems like the problem comes from the psyco_adapters dict shared by all interpreters: register_adapter(datetime.date, adapt_date) call in register_adapters_and_converters() of Lib/sqlite3/dbapi2.py.

Traceback when the bug occurs:

(gdb) py-bt
Traceback (most recent call first):
  <built-in method register_adapter of module object at remote 0x7fffd8118950>
  File "/home/vstinner/python/3.10/Lib/sqlite3/dbapi2.py", line 80, in register_adapters_and_converters
    register_adapter(datetime.date, adapt_date)
  File "/home/vstinner/python/3.10/Lib/sqlite3/dbapi2.py", line 85, in <module>
    register_adapters_and_converters()
  <built-in method exec of module object at remote 0x7fffe9bd5d30>
  (...)
  File "/home/vstinner/python/3.10/Lib/sqlite3/__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  <built-in method exec of module object at remote 0x7fffe9bd5d30>
  (...)
  File "<string>", line 2, in <module>
  <built-in method run_in_subinterp of module object at remote 0x7fffea4a72f0>
  File "/home/vstinner/python/3.10/sqlite3_crash.py", line 12, in doIt
    _testcapi.run_in_subinterp(code)
  (...)

(gdb) frame 10
#10 0x00007fffe80c690d in pysqlite_register_adapter_impl (module=<module at remote 0x7fffd8118950>, type=0x7fffe80f3320 <PyDateTime_DateType>, 
    caster=<function at remote 0x7fffe9bf7330>) at /home/vstinner/python/3.10/Modules/_sqlite/module.c:175
175	    rc = pysqlite_microprotocols_add(type, (PyObject*)pysqlite_PrepareProtocolType, caster);

(gdb) frame 9
#9  0x00007fffe80c5dda in pysqlite_microprotocols_add (type=0x7fffe80f3320 <PyDateTime_DateType>, proto=<type at remote 0x7fffa80a9790>, 
    cast=<function at remote 0x7fffe9bf7330>) at /home/vstinner/python/3.10/Modules/_sqlite/microprotocols.c:68
68	    rc = PyDict_SetItem(psyco_adapters, key, cast);

(gdb) info threads
  Id   Target Id                                   Frame 
  1    Thread 0x7ffff7c7b740 (LWP 332120) "python" 0x00007ffff7d1f3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6
  3    Thread 0x7fffe9af8640 (LWP 332122) "python" 0x00007ffff7d1f3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6
  5    Thread 0x7fffe8af6640 (LWP 332124) "python" 0x00007ffff7d1f3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6
* 6    Thread 0x7fffdbfff640 (LWP 332125) "python" 0x00007ffff7d2454c in __pthread_kill_implementation () from /lib64/libc.so.6
  9    Thread 0x7fffda7fc640 (LWP 332128) "python" 0x00007ffff7d1f3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6
  10   Thread 0x7fffd9ffb640 (LWP 332129) "python" 0x00007ffff7d1f3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6
  16   Thread 0x7fffb67fc640 (LWP 332135) "python" 0x00007ffff7d1f3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6

[vstinner]

In the 3.10 branch, the script doesn't crash before my change

Well, gc_fini_untrack() solves one problem but not all of them :-( I wrote PR #92037 to fix this issue.

Explanation of this bug:

• (1) Interpreter A is created
• (2) Interpreter A creates an adapt_date() function and stores it in sqlite3 adapters using _sqlite3.register_adapter() function
• (3) Interpreter B is created
• (4) Interpreter B does the same: create an adapt_date() function and overrides the sqlite3 adapters registered by interperter A using _sqlite3.register_adapter() function
• (5) Interpreter A is deleted
• (6) Interpreter B is deleted

In this order, everything is fine: the interpreter A still exists when the interpreter B overrides the datetime adapter.

The problem is when the interpreter A is deleted before the interpreter B registers is its own adapter (override the key in the shared dict). In this case, the function is untracked by the GC. Problem: deleting a function requires the function to be tracked by the GC. Othserwise, _PyObject_GC_UNTRACK() assertion fails: this function must only be called on objects called by the GC.

[vstinner]

By the way, when the bug occurs, Python built in debug mode displays this error:

$ ./python sqlite3_crash.py 
Objects/funcobject.c:641: _PyObject_GC_UNTRACK: Assertion "(((PyGC_Head *)(op)-1)->_gc_next != 0)" failed: object not tracked by the garbage collector
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7faa31e94730
object refcount : 0
object type     : 0x88d660
object type name: function
object repr     : 

But then hangs: _PyObject_Dump() tries to acquire the GIL, but another interpreter is holding it...

[vstinner]

cc @ericsnowcurrently

[vstinner]

I got the confirmation that my change fix Kodi crash: #90619 (comment)

[vstinner]

Fixed by:

• main: 1424336
• 3.10: 178a238
• 3.9: f84c51e

[vstinner]

I marked #90619 "Kodi crashing" as a duplicate of this bug.

[matthuisman]

@vstinner
Great work.

Do you know what versions of 3.9 and 3.10 will include this fix?

Oh. Looks like 3.10.5 in a month

[vstinner]

Do you know what versions of 3.9 and 3.10 will include this fix?

The next releases. See:

• 3.9: https://peps.python.org/pep-0596/
• 3.10: https://peps.python.org/pep-0619/

[vstinner]

If you are blocked, you might try to detect which deleted objects are causing crashes and keep a copy of them in a list somewhere: #92036 (comment) Or even somehow call Py_IncRef() on them.