Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DocDB] e6bb62ab80a9c419df8a2807e2fe182cd2d7aa58/D34307 introduced a TSAN race #22101

Closed
1 task done
hari90 opened this issue Apr 22, 2024 · 0 comments
Closed
1 task done

[DocDB] e6bb62ab80a9c419df8a2807e2fe182cd2d7aa58/D34307 introduced a TSAN race #22101

hari90 opened this issue Apr 22, 2024 · 0 comments
Assignees
@hari90
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature kind/failing-test Tests and testing infra priority/medium Medium priority issue status/awaiting-triage Issue awaiting triage

Comments

@hari90
Copy link
Contributor

hari90 commented Apr 22, 2024 •
edited by jira bot
Loading

Jira Link: DB-11034

Description

==================
WARNING: ThreadSanitizer: data race (pid=685702)
Write of size 4 at 0x7b6000005328 by main thread:
#0 boost::asio::ip::address::operator=(boost::asio::ip::address const&) ${YB_THIRDPARTY_DIR}/installed/tsan/include/boost/asio/ip/impl/address.ipp:74:9 (libserver_process.so+0x9415f)
#1 yb::rpc::Messenger::TEST_SetOutboundIpBase(boost::asio::ip::address const&) ${YB_SRC_ROOT}/src/yb/rpc/messenger.h:299:28 (libserver_process.so+0x9415f)
#2 yb::server::TEST_SetupConnectivity(yb::rpc::Messenger*, unsigned long) ${YB_SRC_ROOT}/src/yb/server/server_base.cc:779:14 (libserver_process.so+0x9415f)
#3 yb::tserver::MiniTabletServer::Reconnect() ${YB_SRC_ROOT}/src/yb/tserver/mini_tablet_server.cc:163:3 (libtserver_test_util.so+0x24eb4)
#4 yb::tserver::MiniTabletServer::Start(yb::StronglyTypedBoolyb::tserver::WaitTabletsBootstrapped_Tag) ${YB_SRC_ROOT}/src/yb/tserver/mini_tablet_server.cc:149:3 (libtserver_test_util.so+0x24c58)
#5 yb::MiniCluster::AddTabletServer(yb::tserver::TabletServerOptions const&) ${YB_SRC_ROOT}/src/yb/integration-tests/mini_cluster.cc:387:3 (libintegration-tests.so+0x19b8df)
#6 yb::MiniCluster::AddTabletServer() ${YB_SRC_ROOT}/src/yb/integration-tests/mini_cluster.cc:396:10 (libintegration-tests.so+0x19bc80)
#7 yb::MiniCluster::StartAsync(std::vector<yb::tserver::TabletServerOptions, std::allocatoryb::tserver::TabletServerOptions> const&) ${YB_SRC_ROOT}/src/yb/integration-tests/mini_cluster.cc:228:9 (libintegration-tests.so+0x1986ab)
#8 yb::MiniCluster::Start(std::vector<yb::tserver::TabletServerOptions, std::allocatoryb::tserver::TabletServerOptions> const&) ${YB_SRC_ROOT}/src/yb/integration-tests/mini_cluster.cc:309:3 (libintegration-tests.so+0x19bfda)
#9 yb::client::ClientTest::SetUp() ${YB_SRC_ROOT}/src/yb/client/client-test.cc:198:5 (client-test+0x1cecc3)
#10 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) ${YB_THIRDPARTY_DIR}/src/googletest-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x894f9)
#11 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) ${YB_THIRDPARTY_DIR}/src/googletest-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x894f9)
#12 testing::Test::Run() ${YB_THIRDPARTY_DIR}/src/googletest-1.12.1/googletest/src/gtest.cc:2669:3 (libgtest.so.1.12.1+0x611b7)
#13 testing::TestInfo::Run() ${YB_THIRDPARTY_DIR}/src/googletest-1.12.1/googletest/src/gtest.cc:2853:11 (libgtest.so.1.12.1+0x62a05)
#14 testing::TestSuite::Run() ${YB_THIRDPARTY_DIR}/src/googletest-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x63f04)
#15 testing::internal::UnitTestImpl::RunAllTests() ${YB_THIRDPARTY_DIR}/src/googletest-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x7be3d)
#16 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) ${YB_THIRDPARTY_DIR}/src/googletest-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x8a7cd)
#17 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) ${YB_THIRDPARTY_DIR}/src/googletest-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x8a7cd)
#18 testing::UnitTest::Run() ${YB_THIRDPARTY_DIR}/src/googletest-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x7b5bd)
#19 RUN_ALL_TESTS() ${YB_THIRDPARTY_DIR}/installed/tsan/include/gtest/gtest.h:2293:73 (libyb_test_main.so+0x49e9)
#20 main ${YB_SRC_ROOT}/src/yb/util/test_main.cc:110:13 (libyb_test_main.so+0x49e9)

Previous read of size 4 at 0x7b6000005328 by thread T150:
#0 boost::asio::ip::address::is_unspecified() const ${YB_THIRDPARTY_DIR}/installed/tsan/include/boost/asio/ip/impl/address.ipp:203:11 (libyrpc.so+0x141107)
#1 yb::rpc::Reactor::FindOrStartConnection(yb::rpc::ConnectionId const&, string const&, std::shared_ptryb::rpc::Connection) ${YB_SRC_ROOT}/src/yb/rpc/reactor.cc:760:42 (libyrpc.so+0x141107)
#2 yb::rpc::Reactor::AssignOutboundCall(std::shared_ptryb::rpc::OutboundCall const&) ${YB_SRC_ROOT}/src/yb/rpc/reactor.cc:560:14 (libyrpc.so+0x140738)
#3 yb::rpc::Reactor::ProcessOutboundQueue() ${YB_SRC_ROOT}/src/yb/rpc/reactor.cc:903:17 (libyrpc.so+0x13990b)
#4 decltype(std::declvalyb::rpc::Reactor*&().std::declval<void (yb::rpc::Reactor::&)()>()()) std::__invoke[abi:v170002]<void (yb::rpc::Reactor::&)(), yb::rpc::Reactor&, void>(void (yb::rpc::Reactor::&)(), yb::rpc::Reactor&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__type_traits/invoke.h:308:25 (libyrpc.so+0x14df6b)
#5 std::__bind_return<void (yb::rpc::Reactor::)(), std::tupleyb::rpc::Reactor*, std::tupleyb::rpc::Reactor*&, __is_valid_bind_return<void (yb::rpc::Reactor::)(), std::tupleyb::rpc::Reactor*, std::tupleyb::rpc::Reactor*&>::value>::type std::__apply_functor[abi:v170002]<void (yb::rpc::Reactor::)(), std::tupleyb::rpc::Reactor*, 0ul, std::tupleyb::rpc::Reactor*&>(void (yb::rpc::Reactor::&)(), std::tupleyb::rpc::Reactor*&, std::__tuple_indices<0ul>, std::tupleyb::rpc::Reactor*&&&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:260:12 (libyrpc.so+0x14df6b)
#6 std::__bind_return<void (yb::rpc::Reactor::)(), std::tupleyb::rpc::Reactor*, std::tupleyb::rpc::Reactor*&, __is_valid_bind_return<void (yb::rpc::Reactor::)(), std::tupleyb::rpc::Reactor*, std::tupleyb::rpc::Reactor*&>::value>::type std::__bind<void (yb::rpc::Reactor::)(), yb::rpc::Reactor>::operator()[abi:v170002]yb::rpc::Reactor*&(yb::rpc::Reactor*&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:292:20 (libyrpc.so+0x14df6b)
#7 yb::rpc::FunctorReactorTask<std::__bind<void (yb::rpc::Reactor::)(), yb::rpc::Reactor>>::Run(yb::rpc::Reactor*) ${YB_SRC_ROOT}/src/yb/rpc/reactor_task.h:92:5 (libyrpc.so+0x14df6b)
#8 yb::rpc::Reactor::AsyncHandler(ev::async&, int) ${YB_SRC_ROOT}/src/yb/rpc/reactor.cc:545:11 (libyrpc.so+0x14014d)
#9 void ev::base<ev_async, ev::async>::method_thunk<yb::rpc::Reactor, &yb::rpc::Reactor::AsyncHandler(ev::async&, int)>(ev_loop*, ev_async*, int) ${YB_THIRDPARTY_DIR}/installed/common/include/ev++.h:479:7 (libyrpc.so+0x14eac8)
#10 ev_invoke_pending (libev.so.4+0x871a)
#11 decltype(std::declvalyb::rpc::Reactor*&().std::declval<void (yb::rpc::Reactor::&)()>()()) std::__invoke[abi:v170002]<void (yb::rpc::Reactor::&)(), yb::rpc::Reactor*&, void>(void (yb::rpc::Reactor::&)(), yb::rpc::Reactor&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__type_traits/invoke.h:308:25 (libyrpc.so+0x14eceb)
#12 std::__bind_return<void (yb::rpc::Reactor::)(), std::tupleyb::rpc::Reactor*, std::tuple<>, __is_valid_bind_return<void (yb::rpc::Reactor::)(), std::tupleyb::rpc::Reactor*, std::tuple<>>::value>::type std::__apply_functor[abi:v170002]<void (yb::rpc::Reactor::)(), std::tupleyb::rpc::Reactor*, 0ul, std::tuple<>>(void (yb::rpc::Reactor::&)(), std::tupleyb::rpc::Reactor*&, std::__tuple_indices<0ul>, std::tuple<>&&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:260:12 (libyrpc.so+0x14eceb)
#13 std::__bind_return<void (yb::rpc::Reactor::)(), std::tupleyb::rpc::Reactor*, std::tuple<>, __is_valid_bind_return<void (yb::rpc::Reactor::)(), std::tupleyb::rpc::Reactor*, std::tuple<>>::value>::type std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>::operator()[abi:v170002]<>() ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:292:20 (libyrpc.so+0x14eceb)
#14 decltype(std::declval<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&>()()) std::__invoke[abi:v170002]<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&>(std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__type_traits/invoke.h:340:25 (libyrpc.so+0x14eceb)
#15 void std::__invoke_void_return_wrapper<void, true>::__call[abi:v170002]<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&>(std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__type_traits/invoke.h:415:5 (libyrpc.so+0x14eceb)
#16 std::__function::__alloc_func<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>, std::allocator<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>>, void ()>::operator()abi:v170002 ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/function.h:192:16 (libyrpc.so+0x14eceb)
#17 std::__function::__func<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>, std::allocator<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>>, void ()>::operator()() ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/function.h:363:12 (libyrpc.so+0x14eceb)
#18 std::__function::__value_func<void ()>::operator()abi:v170002 const ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/function.h:517:16 (libyb_util.so+0x41720f)
#19 std::function<void ()>::operator()() const ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1168:12 (libyb_util.so+0x41720f)
#20 yb::Thread::SuperviseThread(void*) ${YB_SRC_ROOT}/src/yb/util/thread.cc:866:3 (libyb_util.so+0x41720f)

Issue Type

kind/failing-test

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.
@hari90 hari90 added area/docdb YugabyteDB core features status/awaiting-triage Issue awaiting triage labels Apr 22, 2024
@hari90 hari90 self-assigned this Apr 22, 2024
@yugabyte-ci yugabyte-ci added kind/failing-test Tests and testing infra priority/medium Medium priority issue kind/enhancement This is an enhancement of an existing feature labels Apr 22, 2024
@hari90 hari90 closed this as completed in b47e512 Apr 22, 2024
svarnau pushed a commit that referenced this issue May 25, 2024
@hari90
[#22101] DocDB: Don't set TEST_SetOutboundIpBase twice
869e0fc 
Summary:
e6bb62a/D34307 introduced a TSAN race.

Setting TEST_SetOutboundIpBase from the test causing a TSAN race.
This function is not required to be called anymore from `TEST_SetupConnectivity` since the server sets the OutboundIpBase on its messenger as part of process startup.

Fixes #22101
Jira: DB-11034

Test Plan:
Jenkins Ready
Jenkins Urgent

Reviewers: esheng, asrivastava

Reviewed By: esheng

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D34394
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hari90 hari90

Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature kind/failing-test Tests and testing infra priority/medium Medium priority issue status/awaiting-triage Issue awaiting triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hari90 @yugabyte-ci