Skip to content

Commit

Permalink
Add workaround for Python threading bug (DM-38669)
Browse files Browse the repository at this point in the history
The bug causes occasional issues with multi-process pipeline executions
with pytest using concurrent running. This is a known issue
(python/cpython#102512) which will eventually
be fixed in Python, at that time the patch should become no-op and can
be removed.
  • Loading branch information
andy-slac committed Apr 12, 2023
1 parent dd9b229 commit 6bdedea
Showing 1 changed file with 10 additions and 0 deletions.
10 changes: 10 additions & 0 deletions python/lsst/ctrl/mpexec/mpGraphExecutor.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
import pickle
import signal
import sys
import threading
import time
from collections.abc import Iterable
from enum import Enum
Expand Down Expand Up @@ -141,6 +142,15 @@ def _executeJob(
snd_conn : `multiprocessing.Connection`
Connection to send job report to parent process.
"""
# This terrible hack is a workaround for Python threading bug:
# https://github.com/python/cpython/issues/102512. Should be removed
# when fix for that bug is deployed. Inspired by
# https://github.com/QubesOS/qubes-core-admin-client/pull/236/files.
thread = threading.current_thread()
if isinstance(thread, threading._DummyThread):
if getattr(thread, "_tstate_lock", "") is None:
thread._set_tstate_lock() # type: ignore[attr-defined]

if logConfigState and not CliLog.configState:
# means that we are in a new spawned Python process and we have to
# re-initialize logging
Expand Down

0 comments on commit 6bdedea

Please sign in to comment.