Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[YSQL] Multiple YSQL tests fail when TSAN enabled due to race in postgres shutdown path #15938

Closed
deeps1991 opened this issue Feb 1, 2023 · 0 comments
Closed

[YSQL] Multiple YSQL tests fail when TSAN enabled due to race in postgres shutdown path #15938

deeps1991 opened this issue Feb 1, 2023 · 0 comments
Assignees
@deeps1991
Labels
area/ysql Yugabyte SQL (YSQL) kind/bug This issue is a bug priority/medium Medium priority issue

Comments

@deeps1991
Copy link
Contributor

deeps1991 commented Feb 1, 2023 •
edited
Loading

Jira Link: DB-5351

Description

WARNING: ThreadSanitizer: data race (pid=2761)
  Write of size 8 at 0x7fbf5edbf5f0 by main thread:
    #0 YBCDestroyPgGate ${BUILD_ROOT}/../../src/yb/yql/pggate/ybc_pggate.cc:190:11 (libyb_pggate.so+0x7f6ce)
    #1 YBOnPostgresBackendShutdown ${YB_SRC_ROOT}/src/postgres/src/backend/utils/misc/../../../../../../../src/postgres/src/backend/utils/misc/pg_yb_utils.c:620:2 (postgres+0xc1c233)
    #2 proc_exit ${YB_SRC_ROOT}/src/postgres/src/backend/storage/ipc/../../../../../../../src/postgres/src/backend/storage/ipc/ipc.c:152:3 (postgres+0x9bccec)
    #3 errfinish ${YB_SRC_ROOT}/src/postgres/src/backend/utils/error/../../../../../../../src/postgres/src/backend/utils/error/elog.c:579:3 (postgres+0xbdb142)
    #4 YBPreloadRelCache ${YB_SRC_ROOT}/src/postgres/src/backend/utils/cache/../../../../../../../src/postgres/src/backend/utils/cache/relcache.c:1953:3 (postgres+0xbb5474)
    #5 YbPreloadRelCacheHelper ${YB_SRC_ROOT}/src/postgres/src/backend/tcop/../../../../../../src/postgres/src/backend/tcop/postgres.c:3687:3 (postgres+0xa0505a)
    #6 YBRefreshCache ${YB_SRC_ROOT}/src/postgres/src/backend/tcop/../../../../../../src/postgres/src/backend/tcop/postgres.c:3745:2 (postgres+0xa04e46)
    #7 YBCheckSharedCatalogCacheVersion ${YB_SRC_ROOT}/src/postgres/src/backend/tcop/../../../../../../src/postgres/src/backend/tcop/postgres.c:4042:3 (postgres+0x9fffea)
    #8 PostgresMain ${YB_SRC_ROOT}/src/postgres/src/backend/tcop/../../../../../../src/postgres/src/backend/tcop/postgres.c:5222:4 (postgres+0x9fffea)
    #9 BackendRun ${YB_SRC_ROOT}/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4674:2 (postgres+0x925fcf)
    #10 BackendStartup ${YB_SRC_ROOT}/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4312:3 (postgres+0x924f65)
    #11 ServerLoop ${YB_SRC_ROOT}/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1775:7 (postgres+0x924f65)
    #12 PostmasterMain ${YB_SRC_ROOT}/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1431:11 (postgres+0x91fda6)
    #13 PostgresServerProcessMain ${YB_SRC_ROOT}/src/postgres/src/backend/main/../../../../../../src/postgres/src/backend/main/main.c:234:3 (postgres+0x81342e)
    #14 main <null> (postgres+0x813a41)

  Previous read of size 8 at 0x7fbf5edbf5f0 by thread T1:
    #0 YBCPgIsYugaByteEnabled ${BUILD_ROOT}/../../src/yb/yql/pggate/ybc_pggate.cc:1335:10 (libyb_pggate.so+0x8789a)
    #1 IsYugaByteEnabled ${YB_SRC_ROOT}/src/postgres/src/backend/utils/misc/../../../../../../../src/postgres/src/backend/utils/misc/pg_yb_utils.c:173:9 (postgres+0xc1ad65)
    #2 die ${YB_SRC_ROOT}/src/postgres/src/backend/tcop/../../../../../../src/postgres/src/backend/tcop/postgres.c:2732:6 (postgres+0x9fd974)
    #3 __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, int, __sanitizer::__sanitizer_siginfo*, void*) /opt/yb-build/llvm/yb-llvm-v15.0.3-yb-1-1667341687-0b8d1183-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2025:5 (postgres+0x3fe74a)
    #4 void boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock>(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/include/boost/asio/detail/posix_event.hpp:119:7 (libyrpc.so+0xee316)
    #5 boost::asio::detail::conditionally_enabled_event::wait(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/include/boost/asio/detail/conditionally_enabled_event.hpp:97:14 (libyrpc.so+0xee316)
    #6 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/include/boost/asio/detail/impl/scheduler.ipp:501:21 (libyrpc.so+0xee316)
    #7 boost::asio::detail::scheduler::run(boost::system::error_code&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/include/boost/asio/detail/impl/scheduler.ipp:210:10 (libyrpc.so+0xedd61)
    #8 boost::asio::io_context::run(boost::system::error_code&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/include/boost/asio/impl/io_context.ipp:71:16 (libyrpc.so+0xe957e)
    #9 yb::rpc::IoThreadPool::Impl::Execute() ${BUILD_ROOT}/../../src/yb/rpc/io_thread_pool.cc:76:17 (libyrpc.so+0xe957e)
    #10 decltype(*std::declval<yb::rpc::IoThreadPool::Impl*&>().*std::declval<void (yb::rpc::IoThreadPool::Impl::*&)()>()()) std::__invoke[abi:v15003]<void (yb::rpc::IoThreadPool::Impl::*&)(), yb::rpc::IoThreadPool::Impl*&, void>(void (yb::rpc::IoThreadPool::Impl::*&)(), yb::rpc::IoThreadPool::Impl*&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:359:23 (libyrpc.so+0xef74b)
    #11 std::__bind_return<void (yb::rpc::IoThreadPool::Impl::*)(), std::tuple<yb::rpc::IoThreadPool::Impl*>, std::tuple<>, __is_valid_bind_return<void (yb::rpc::IoThreadPool::Impl::*)(), std::tuple<yb::rpc::IoThreadPool::Impl*>, std::tuple<>>::value>::type std::__apply_functor[abi:v15003]<void (yb::rpc::IoThreadPool::Impl::*)(), std::tuple<yb::rpc::IoThreadPool::Impl*>, 0ul, std::tuple<>>(void (yb::rpc::IoThreadPool::Impl::*&)(), std::tuple<yb::rpc::IoThreadPool::Impl*>&, std::__tuple_indices<0ul>, std::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:263:12 (libyrpc.so+0xef74b)
    #12 std::__bind_return<void (yb::rpc::IoThreadPool::Impl::*)(), std::tuple<yb::rpc::IoThreadPool::Impl*>, std::tuple<>, __is_valid_bind_return<void (yb::rpc::IoThreadPool::Impl::*)(), std::tuple<yb::rpc::IoThreadPool::Impl*>, std::tuple<>>::value>::type std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>::operator()[abi:v15003]<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:295:20 (libyrpc.so+0xef74b)
    #13 decltype(std::declval<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>&>()()) std::__invoke[abi:v15003]<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>&>(std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394:23 (libyrpc.so+0xef74b)
    #14 std::__bind_return<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>, std::tuple<>, std::tuple<>, __is_valid_bind_return<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>, std::tuple<>, std::tuple<>>::value>::type std::__apply_functor[abi:v15003]<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>, std::tuple<>, std::tuple<>>(std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>&, std::tuple<>&, std::__tuple_indices<>, std::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:263:12 (libyrpc.so+0xef74b)
    #15 std::__bind_return<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>, std::tuple<>, std::tuple<>, __is_valid_bind_return<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>, std::tuple<>, std::tuple<>>::value>::type std::__bind<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>>::operator()[abi:v15003]<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:295:20 (libyrpc.so+0xef74b)
    #16 decltype(std::declval<std::__bind<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>>&>()()) std::__invoke[abi:v15003]<std::__bind<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>>&>(std::__bind<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394:23 (libyrpc.so+0xef74b)
    #17 void std::__invoke_void_return_wrapper<void, true>::__call<std::__bind<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>>&>(std::__bind<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:479:9 (libyrpc.so+0xef74b)
    #18 std::__function::__alloc_func<std::__bind<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>>, std::allocator<std::__bind<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>>>, void ()>::operator()[abi:v15003]() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185:16 (libyrpc.so+0xef74b)
    #19 std::__function::__func<std::__bind<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>>, std::allocator<std::__bind<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>>>, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:359:12 (libyrpc.so+0xef74b)
    #20 std::__function::__value_func<void ()>::operator()[abi:v15003]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:512:16 (libyb_util.so+0x354e64)
    #21 std::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221115221900-32d6b89b02-centos7-x86_64-clang15/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1197:12 (libyb_util.so+0x354e64)
    #22 yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:800:3 (libyb_util.so+0x354e64)

  Location is global 'yb::pggate::(anonymous namespace)::pgapi' of size 8 at 0x7fbf5edbf5f0 (libyb_pggate.so+0x1315f0)

  Thread T1 'iotp_pggate_ybc' (tid=2764, finished) created by main thread at:
    #0 pthread_create /opt/yb-build/llvm/yb-llvm-v15.0.3-yb-1-1667341687-0b8d1183-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1022:3 (postgres+0x3f7f2d)
    #1 yb::Thread::StartThread(string const&, string const&, std::function<void ()>, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:722:15 (libyb_util.so+0x3535da)
    #2 yb::Result<scoped_refptr<yb::Thread>> yb::Thread::Make<std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>>(string const&, string const&, std::__bind<void (yb::rpc::IoThreadPool::Impl::*)(), yb::rpc::IoThreadPool::Impl*>&&) ${BUILD_ROOT}/../../src/yb/util/thread.h:202:5 (libyrpc.so+0xe93af)
    #3 yb::rpc::IoThreadPool::Impl::Impl(string const&, unsigned long) ${BUILD_ROOT}/../../src/yb/rpc/io_thread_pool.cc:38:26 (libyrpc.so+0xe87bd)
    #4 yb::rpc::IoThreadPool::IoThreadPool(string const&, unsigned long) ${BUILD_ROOT}/../../src/yb/rpc/io_thread_pool.cc:87:17 (libyrpc.so+0xe83e7)
    #5 yb::rpc::Messenger::Messenger(yb::rpc::MessengerBuilder const&) ${BUILD_ROOT}/../../src/yb/rpc/messenger.cc:554:7 (libyrpc.so+0xf636c)
    #6 yb::rpc::MessengerBuilder::Build() ${BUILD_ROOT}/../../src/yb/rpc/messenger.cc:149:44 (libyrpc.so+0xf0e97)
    #7 yb::client::CreateClientMessenger(string const&, int, scoped_refptr<yb::MetricEntity> const&, std::shared_ptr<yb::MemTracker> const&, yb::rpc::SecureContext*) ${BUILD_ROOT}/../../src/yb/client/client_utils.cc:70:20 (libyb_client.so+0x3a4798)
    #8 yb::pggate::(anonymous namespace)::BuildMessenger(string const&, int, scoped_refptr<yb::MetricEntity> const&, std::shared_ptr<yb::MemTracker> const&) ${BUILD_ROOT}/../../src/yb/yql/pggate/pggate.cc:136:20 (libyb_pggate.so+0x8b635)
    #9 yb::pggate::PgApiContext::PgApiContext() ${BUILD_ROOT}/../../src/yb/yql/pggate/pggate.cc:335:24 (libyb_pggate.so+0x8b635)
    #10 yb::pggate::YBCInitPgGateEx(PgTypeEntity const*, int, PgCallbacks, yb::pggate::PgApiContext*) ${BUILD_ROOT}/../../src/yb/yql/pggate/ybc_pggate.cc:172:35 (libyb_pggate.so+0x7f403)
    #11 YBCInitPgGate ${BUILD_ROOT}/../../src/yb/yql/pggate/ybc_pggate.cc:180:3 (libyb_pggate.so+0x7f667)
    #12 YBInitPostgresBackend ${YB_SRC_ROOT}/src/postgres/src/backend/utils/misc/../../../../../../../src/postgres/src/backend/utils/misc/pg_yb_utils.c:604:3 (postgres+0xc1c1af)
    #13 InitPostgresImpl ${YB_SRC_ROOT}/src/postgres/src/backend/utils/init/../../../../../../../src/postgres/src/backend/utils/init/postinit.c (postgres+0xbf522d)
    #14 InitPostgres ${YB_SRC_ROOT}/src/postgres/src/backend/utils/init/../../../../../../../src/postgres/src/backend/utils/init/postinit.c:1179:3 (postgres+0xbf500b)
    #15 PostgresMain ${YB_SRC_ROOT}/src/postgres/src/backend/tcop/../../../../../../src/postgres/src/backend/tcop/postgres.c:4861:2 (postgres+0x9ff29c)
    #16 BackendRun ${YB_SRC_ROOT}/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4674:2 (postgres+0x925fcf)
    #17 BackendStartup ${YB_SRC_ROOT}/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:4312:3 (postgres+0x924f65)
    #18 ServerLoop ${YB_SRC_ROOT}/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1775:7 (postgres+0x924f65)
    #19 PostmasterMain ${YB_SRC_ROOT}/src/postgres/src/backend/postmaster/../../../../../../src/postgres/src/backend/postmaster/postmaster.c:1431:11 (postgres+0x91fda6)
    #20 PostgresServerProcessMain ${YB_SRC_ROOT}/src/postgres/src/backend/main/../../../../../../src/postgres/src/backend/main/main.c:234:3 (postgres+0x81342e)
    #21 main <null> (postgres+0x813a41)

SUMMARY: ThreadSanitizer: data race ${BUILD_ROOT}/../../src/yb/yql/pggate/ybc_pggate.cc:190:11 in YBCDestroyPgGate
@deeps1991 deeps1991 added area/ysql Yugabyte SQL (YSQL) status/awaiting-triage Issue awaiting triage labels Feb 1, 2023
@deeps1991 deeps1991 self-assigned this Feb 1, 2023
@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels Feb 1, 2023
@deeps1991 deeps1991 removed kind/bug This issue is a bug priority/medium Medium priority issue status/awaiting-triage Issue awaiting triage labels Feb 1, 2023
@deeps1991 deeps1991 changed the title [YSQL] Multiple YSQL tests fail when TSAN enabled due to races in postgres shutdown path [YSQL] Multiple YSQL tests fail when TSAN enabled due to race in postgres shutdown path Feb 1, 2023
@deeps1991 deeps1991 mentioned this issue Feb 1, 2023
Open
deeps1991 added a commit that referenced this issue Feb 21, 2023
@deeps1991
[#12884, #15938] YSQL: Fix TSAN issues caused by quickdie()
c5f2231 
Summary:
quickdie() is the signal handler invoked to handle SIGQUIT. This is the
immediate shutdown mode, expected to be invoked when something is really
wrong, such as corrupted shared memory.

This being the case, it is not safe to invoke
YBOnPostgresBackendShutdown, which internally invokes PgApi destructor
from quickdie. This also causes TSAN failures because there could be
other concurrent threads in process of accessing the PgApi object.

Secondly as noted in the comment in quickdie(), it is not safe to call
ereport() from quickdie as it is not async-signal-safe. It has been
added by PG authors anyway as a best effort to provide some information before shutting
down. Although acceptable, this again causes TSAN failures. Therefore
this patch silences the TSAN failures by not invoking the ereport() in
TSAN build.

Test Plan: Manually tested by running tests enabled with TSAN.

Reviewers: dmitry

Reviewed By: dmitry

Subscribers: yql

Differential Revision: https://phabricator.dev.yugabyte.com/D23045
@deeps1991 deeps1991 mentioned this issue Apr 4, 2023
Closed
1 task
@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels Jun 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@deeps1991 deeps1991

Labels
area/ysql Yugabyte SQL (YSQL) kind/bug This issue is a bug priority/medium Medium priority issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yugabyte-ci @deeps1991