[YSQL] lock-order-inversion observed in pgwrapper::PgSupervisor #16055
Labels
Comments
deeps1991
added
area/ysql
yugabyte-ci
added
kind/bug
|
Start can only be called once so we can never actually deadlock on this. Can we suppress this? |
deeps1991
added a commit
that referenced
this issue
Jun 8, 2023
Summary: TSAN detects this possible deadlock in PgWrapper::Supervisor: Minimal stack trace: ``` WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=8513) [ts-2] Cycle in lock order graph: M0 (0x7b5400002a48) => M1 (0x7b1800051f18) => M0 Mutex M1 acquired here while holding mutex M0 in main thread: [ts-2] #0 pthread_mutex_lock /opt/yb-build/llvm/yb-llvm-v15.0.3-yb-1-1667341687-0b8d1183-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:4457:3 (yb-tserver+0xa4cfa) [ts-2] #3 yb::FlagsCallbackRegistry::RegisterCallback(void const*, string const&, std::function<void ()>) ${BUILD_ROOT}/../../src/yb/util/flags/flags_callback.cc:84:19 (libyb_util.so+0x24dcd9) [ts-2] #4 yb::RegisterFlagUpdateCallback(void const*, string const&, std::function<void ()>) ${BUILD_ROOT}/../../src/yb/util/flags/flags_callback.cc:184:24 (libyb_util.so+0x24f8c3) [ts-2] #5 yb::pgwrapper::PgSupervisor::RegisterReloadPgConfigCallback(void const*) ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/pg_wrapper.cc:904:32 (libyb_pgwrapper.so+0x2f2f4) [ts-2] #6 yb::pgwrapper::PgSupervisor::RegisterPgFlagChangeNotifications() ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/pg_wrapper.cc:919:7 (libyb_pgwrapper.so+0x2e69e) [ts-2] #7 yb::pgwrapper::PgSupervisor::Start() ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/pg_wrapper.cc:749:3 (libyb_pgwrapper.so+0x2ce05) Mutex M0 previously acquired by the same thread here: [ts-2] #0 pthread_mutex_lock /opt/yb-build/llvm/yb-llvm-v15.0.3-yb-1-1667341687-0b8d1183-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:4457:3 (yb-tserver+0xa4cfa) [ts-2] #3 yb::pgwrapper::PgSupervisor::Start() ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/pg_wrapper.cc:746:31 (libyb_pgwrapper.so+0x2cdaf) Mutex M0 acquired here while holding mutex M1 in thread T66: [ts-2] #0 pthread_mutex_lock /opt/yb-build/llvm/yb-llvm-v15.0.3-yb-1-1667341687-0b8d1183-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:4457:3 (yb-tserver+0xa4cfa) [ts-2] #3 yb::pgwrapper::PgSupervisor::UpdateAndReloadConfig() ${BUILD_ROOT}/../../src/yb/yql/pgwrapper/pg_wrapper.cc:894:31 (libyb_pgwrapper.so+0x2f1b7) ``` TSAN detects that there are two functions (Start() and UpdateAndReloadConfig) each acquiring M0 and M1 in inverse order which may run into a deadlock. However, Start() is always called first and will acquire M0 and M1 before it registers the callback that invokes UpdateAndReloadConfig(). Start() will never be called again. Thus the deadlock called out by TSAN is not possible. Jira: DB-5450 Test Plan: Tested with TSAN build in Detective runs of D22913 Reviewers: hsunder, mbautin Reviewed By: hsunder Subscribers: yql Differential Revision: https://phorge.dev.yugabyte.com/D25311
|
It looks like the above fix did not work, the issue can still be observed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-5450
Description
When trying to run YSQL tests on TSAN build, the following TSAN failure was observed in a few tests:
Complete logs can be found in the attached files
PgIndexBackfillTest_Permissions.log
PgWrapperFlagsTest_VerifyGFlagRuntimeTag.log
PgIndexBackfillTest_PgStatProgressCreateIndexPartitioned.log
The text was updated successfully, but these errors were encountered: