ResourceWarning: Object of type XXX is not untracked before destruction

[messense]

While investigating pydantic/pydantic-core#451 I noticed that when running on a debug build of Python, the SchemaSerializer pyclass with gc protocol support raised a warning

ResourceWarning: Object of type pydantic_core._pydantic_core.SchemaSerializer is not untracked before destruction

and caused a segfault on debug build of Python only:

collected 3134 items / 3133 deselected / 1 selected

tests/validators/test_datetime.py
Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350287360) at ./nptl/pthread_kill.c:44
Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
44	./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350287360) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737350287360) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737350287360, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7c97476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c7d7f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c7d71b in __assert_fail_base (fmt=0x7ffff7e32150 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x77bda9 "!_PyErr_Occurred(tstate)", file=0x78fc93 "../Objects/typeobject.c", line=1068, function=<optimized out>)
    at ./assert/assert.c:92
#6  0x00007ffff7c8ee96 in __GI___assert_fail (assertion=assertion@entry=0x77bda9 "!_PyErr_Occurred(tstate)", file=file@entry=0x78fc93 "../Objects/typeobject.c", line=line@entry=1068,
    function=function@entry=0x790510 <__PRETTY_FUNCTION__.30> "type_call") at ./assert/assert.c:101
#7  0x000000000052a1c8 in type_call (type=0xa21640 <_PyExc_ResourceWarning>, args=('Object of type pydantic_core._pydantic_core.SchemaValidator is not untracked before destruction',), kwds=0x0) at ../Objects/typeobject.c:1068
#8  0x00000000004b8b15 in _PyObject_MakeTpCall (tstate=tstate@entry=0xb46058 <_PyRuntime+166328>, callable=callable@entry=<type at remote 0xa21640>, args=args@entry=0x7fffffffbba8, nargs=<optimized out>, keywords=keywords@entry=0x0)
    at ../Objects/call.c:214
#9  0x00000000004b8e91 in _PyObject_VectorcallTstate (tstate=0xb46058 <_PyRuntime+166328>, callable=<type at remote 0xa21640>, args=args@entry=0x7fffffffbba8, nargsf=<optimized out>, nargsf@entry=9223372036854775809,
    kwnames=kwnames@entry=0x0) at ../Include/internal/pycore_call.h:90
#10 0x00000000004b8f4e in PyObject_CallOneArg (func=<optimized out>, arg=<optimized out>) at ../Objects/call.c:376

#11 0x000000000058e1fd in warn_explicit (tstate=0xb46058 <_PyRuntime+166328>, category=<type at remote 0xa21640>,
    category@entry=<error reading variable: Dwarf Error: Cannot find DIE at 0x1c9 referenced from DIE at 0x3de638 [in module /root/code/pydantic-core/pydantic_core/_pydantic_core.cpython-311-x86_64-linux-gnu.so]>,
    message=message@entry='Object of type pydantic_core._pydantic_core.SchemaValidator is not untracked before destruction', filename=filename@entry='gc', lineno=lineno@entry=0, module=module@entry='gc', registry=0x0, sourceline=0x0,
    source=0x0) at ../Python/_warnings.c:671
#12 0x000000000058fe92 in PyErr_WarnExplicitFormat (category=<type at remote 0xa21640>, filename_str=<optimized out>, lineno=0, module_str=<optimized out>, registry=0x0, format=<optimized out>) at ../Python/_warnings.c:1286

Googling it leads us to apache/arrow#33765 (comment) which mentions PyObject_GC_UnTrack. According to its documentation

Remove the object op from the set of container objects tracked by the collector. Note that PyObject_GC_Track() can be called again on this object to add it back to the set of tracked objects. The deallocator (tp_dealloc handler) should call this for the object before any of the fields used by the tp_traverse handler become invalid.

It seems that we should call it in our generated tp_dealloc handler for pyclasses that use the gc protocol?

[messense]

For example, here's how cython handles it:

• Guard deallocation with PyObject_GC_Untrack cython/cython#1352
• https://github.com/cython/cython/blob/b8b378712bc74627f266f3ba6c13751af8e1991f/Cython/Compiler/ModuleNode.py#L1723-L1727
• cython/cython@9a08ff2

[samuelcolvin]

Also worth noting that according to apache/arrow#33765 (comment) the warning is new in 3.11, so you need 3.11-dbg to see it.

[davidhewitt]

I took a look at this briefly today and agree with @messense that the solution is to call PyObject_GC_Untrack at the right time in tp_dealloc.

It's not immediately obvious to me how to get it right in all cases. I also think that while we're revisiting this code we can resume looking at #2479. Rather than try to rush a fix for 0.19 I think we should release without fixing this, and make this a priority fix for 0.20.