"mem" and "object" Allocators are No Longer Protected by the GIL

[ericsnowcurrently]

Once we moved to per-interpreter GIL, the promises in the docs no longer hold:

...where the allocation must be performed with the GIL held.

It's still fine for pymalloc, but any custom, non-wrapping "mem"/"object" allocators would need to be updated to be thread-safe or per-interpreter.

I have a PR up that does an okay job of adapting such allocators: gh-105619. However, it penalizes use of such an allocator in subinterpreters that have their own GIL.

Honestly, I'm leaning toward documenting that such allocators must be thread-safe or per-interpreter. From what I understand, the documented guarantees (in the docs and in PEP 445) are more about representing what pymalloc needs than what custom allocators need.

Perhaps the biggest question is: what projects would be impacted? I haven't had a chance yet to search for projects that use custom mem/object allocators that aren't thread-safe. I suspect there aren't more than two or three.

Linked PRs

• gh-105766: Document that Custom Allocators Must Be Thread-Safe #107519
• [3.12] gh-105766: Document that Custom Allocators Must Be Thread-Safe (gh-107519) #107522
• gh-105766: Clarify About Custom Allocators in C-API Docs #109035

[encukou]

Can we add a flag, so that e.g. PyMem_SetAllocator(PYMEM_DOMAIN_MEM | PyMEM_THREADSAFE, ...) would set PYMEM_DOMAIN_MEM without the wrappers?

[ericsnowcurrently]

python-dev thread: https://discuss.python.org/t/what-to-do-about-custom-allocators-and-the-gil/30327

[ericsnowcurrently]

I'm going to add a note in the docs that custom allocators (that have their own state), including hooks/wrappers, must be thread-safe.

We could also qualify that to only when subinterpreters are used, but I'm not sure that matters. If we did qualify it that way, it may also make sense to add the flag @encukou suggested above (and its inverse).