Crash in Py_Initialize in non-main thread in free-threading build
#123022
Comments
colesbury
added
topic-free-threading
type-crash
|
The part about this not happening in a debug build makes sense. The crashing code path is disabled in debug builds (where Lines 117 to 120 in f84754b I think this might have to do with re-initializing Python (i.e., |
|
This is the relevant code in PyO3 that calls the C API: When I run the test binary in a debugger, there's only one call to |
|
Does this affect all calls to |
This comment was marked as outdated.
This comment was marked as outdated.
Line 46 above is not inside an |
|
The crash occurs when Python is initialized for the first time outside of the main thread. Here's a reproducer: #include <Python.h>
#include <pthread.h>
void *thread_main(void *unused)
{
Py_InitializeEx(0); // crashes!
}
int main()
{
pthread_t thread;
pthread_create(&thread, NULL, thread_main, NULL);
pthread_join(thread, NULL);
return 0;
}The problem is that mimalloc is calling This doesn't happen in the main thread because the main thread's default heap is initialized when the process starts. I think the crash can also occur if the program allocates a lot of memory in total, because the condition is Lines 114 to 115 in d7a3df9 We can fix this by calling |
colesbury
changed the title
Py_Initialize on macOS with PyO3 in free-threading buildPy_Initialize in non-main thread in free-threading build
Check that the current default heap is initialized in `_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`. The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an initialized default heap. This is true for our main thread, but not for background threads. The problematic code path is usually called during initialization (i.e., `Py_Initialize`), but it may also be called if the program allocates large amounts of memory in total. The crash only affected the free-threaded build.
Check that the current default heap is initialized in `_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`. The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an initialized default heap. This is true for our main thread, but not for background threads. The problematic code path is usually called during initialization (i.e., `Py_Initialize`), but it may also be called if the program allocates large amounts of memory in total. The crash only affected the free-threaded build.
…ythonGH-123052) Check that the current default heap is initialized in `_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`. The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an initialized default heap. This is true for our main thread, but not for background threads. The problematic code path is usually called during initialization (i.e., `Py_Initialize`), but it may also be called if the program allocates large amounts of memory in total. The crash only affected the free-threaded build. (cherry picked from commit d061ffe) Co-authored-by: Sam Gross <[email protected]>
…GH-123052) (#123114) Check that the current default heap is initialized in `_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`. The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an initialized default heap. This is true for our main thread, but not for background threads. The problematic code path is usually called during initialization (i.e., `Py_Initialize`), but it may also be called if the program allocates large amounts of memory in total. The crash only affected the free-threaded build. (cherry picked from commit d061ffe) Co-authored-by: Sam Gross <[email protected]>
…ython#123052) Check that the current default heap is initialized in `_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`. The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an initialized default heap. This is true for our main thread, but not for background threads. The problematic code path is usually called during initialization (i.e., `Py_Initialize`), but it may also be called if the program allocates large amounts of memory in total. The crash only affected the free-threaded build.
…ython#123052) Check that the current default heap is initialized in `_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`. The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an initialized default heap. This is true for our main thread, but not for background threads. The problematic code path is usually called during initialization (i.e., `Py_Initialize`), but it may also be called if the program allocates large amounts of memory in total. The crash only affected the free-threaded build.
Py_Initializewill crash if in the free-threaded build if it's not called from the main thread. Additionally, background threads may crash if the program allocates lots of memory in total (>=30 TiB, not necessarily at one time.)The problem is related to our use of mimalloc in the free-threaded build. We don't set mimalloc's "default heap" for threads, but a few code paths assume that there is a default heap.
Originally posted by @ngoldbaum in #122918 (comment)
Both the debug python build and the optimized, crashing Python are built from source using pyenv.
Linked PRs
Py_Initializein background thread #123052Py_Initializein background thread (GH-123052) #123114The text was updated successfully, but these errors were encountered: