PyThreadState_Swap() During Finalization Causes Immediate Exit (AKA Daemon Threads Are Still the Worst!)

[ericsnowcurrently]

Bug report

tl;dr Switching between interpreters while finalizing causes the main thread to exit. The fix should be simple.

We use PyThreadState_Swap() to switch between interpreters. That function almost immediately calls _PyEval_AcquireLock(). During finalization, _PyEval_AcquireLock() immediately causes the thread to exit if the current thread state doesn't match the one that was active when Py_FinalizeEx() was called.

Thus, if we switch interpreters during finalization then the thread will exit. If we do this in the finalizing (main) thread then the process immediately exits with an exit code of 0.

One notable consequence is that a Python process with an unhandled exception will print the traceback like normal but can end up with an exit code of 0 instead of 1 (and some of the runtime finalization code never gets executed). ¹

Reproducer

$ cat > script.py << EOF
import _xxsubinterpreters as _interpreters
interpid = _interpreters.create()
raise Exception
EOF
$ ./python script.py
Traceback (most recent call last):
  File ".../check-swapped-exitcode.py", line 3, in <module>
    raise Exception
Exception
$ echo $?
0

In this case, "interpid" is a PyInterpreterIDObject bound to the __main__ module (of the main interpreter). It is still bound there when the script ends and the executable starts finalizing the runtime by calling Py_FinalizeEx(). ²

Here's what happens in Py_FinalizeEx():

1. wait for non-daemon threads to finish ³
2. run any remaining pending calls belong to the main interpreter
3. run at exit hooks
4. mark the runtime as finalizing (storing the pointer to the current tstate, which belongs to the main interpreter)
5. delete all other tstates belong to the main interpreter (i.e. all daemon threads)
6. remove our custom signal handlers
7. finalize the import state
8. clean up sys.modules of the main interpreter (finalize_modules() in Python/pylifecycle.c)

At the point the following happens:

1. the __main__ module is dealloc'ed
2. "interpid" is dealloc'ed (PyInterpreterID_Type.tp_dealloc)
3. _PyInterpreterState_IDDecref() is called, which finalizes the corresponding interpreter state
   4, before Py_EndInterpreter() is called, we call _PyThreadState_Swap() to switch to a tstate belonging to the subinterpreter
4. that calls _PyEval_AcquireLock()
5. that basically calls _PyThreadState_MustExit(), which sees that the current tstate pointer isn't the one we stored as "finalizing"
6. it then calls PyThread_exit_thread(), which kills the main thread
7. the process exits with an exitcode of 0

Notably, the rest of Py_FinalizeEx() (and Py_Main(), etc.) does not execute. main() never gets a chance to return an exitcode of 1.

Background

Runtime finalization happens in whichever thread called Py_FinalizeEx() and happens relative to whichever PyThreadState is active there. This is typically the main thread and the main interpreter.

Other threads may still be running when we start finalization, whether daemon threads or not, and each of those threads has a thread state corresponding to the interpreter that is active in that thread. ⁴ One of the first things we do during finalization is to wait for all non-daemon threads to finish running. Daemon threads are a different story. They must die!

Back in 2011 we identified that daemon threads were interfering with finalization, sometimes causing crashes or making the Python executable hang. ⁵ At the time, we applied a best-effort solution where we kill the current thread if it isn't the one where Py_FinalizeEx() was called.

However, that solution checked the tstate pointer rather than the thread ID, so swapping interpreters in the finalizing thread was broken, and here we are.

History:

• shutdown (exit) can hang or segfault with daemon threads running #46164 (2011; commit 0d5e52d) - exit thread during finalization in PyEval_RestoreThread() (also add _Py_Finalizing)
• gh-??? (2014; commit 17548dd) - do same in _PyEval_EvalFrameDefault() (eval loop, right after re-acquiring GIL when handling eval breaker)
• PyEval_AcquireLock() and PyEval_AcquireThread() do not handle runtime finalization properly. #80656 (2019; PR: bpo-36475: Finalize PyEval_AcquireLock() and PyEval_AcquireThread() properly #12667) - do same in PyEval_AcquireLock() and PyEval_AcquireThread() (also add exit_thread_if_finalizing())
• Daemon thread is crashing in PyEval_RestoreThread() while the main thread is exiting the process #84058 (2020; PR: bpo-39877: Fix PyEval_RestoreThread() for daemon threads #18811) - use _PyRuntime directly
• Daemon thread is crashing in PyEval_RestoreThread() while the main thread is exiting the process #84058 (2020; PR: bpo-39877: Refactor take_gil() function #18885) - move all the checks to take_gil()

Related: gh-87135 (PRs: gh-105805, gh-28525)

Linked PRs

• gh-109793: Allow Switching Interpreters During Finalization #109794
• [3.12] gh-109793: Allow Switching Interpreters During Finalization (gh-109794) #110705

Footnotes

1. This may help explain why, when we re-run some tests in subprocesses, they aren't marked as failures even when they actually fail. ↩

2. Note that we did not create any extra threads; we stayed exclusively in the main thread. We also didn't even run any code in the subinterpreter. ↩

3. FYI, IIRC we used to abort right before this point if there were any subinterpreters around still. ↩

4. In any given OS thread, each interpreter has a distinct tstate. Each tstate (mostly) corresponds to exactly one OS thread. ↩

5. If a daemon thread keeps running and tries to access any objects or other runtime state then there's a decent chance of a crash. ↩

[ericsnowcurrently]

CC @vstinner, @gpshead