Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[docdb] Tsan race on QLStressTest_LongRemoteBootstrap #2101

Closed
bmatican opened this issue Aug 19, 2019 · 0 comments
Closed

[docdb] Tsan race on QLStressTest_LongRemoteBootstrap #2101

bmatican opened this issue Aug 19, 2019 · 0 comments
Assignees
@ttyusupov
Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug

Comments

@bmatican
Copy link
Contributor

Found in a unit test result: https://detective-gcp.dev.yugabyte.com/job/github-yugabyte-db-phabricator/4534/artifact/build/tsan-clang-dynamic-ninja/yb-test-logs/tests-client__ql-stress-test/QLStressTest_LongRemoteBootstrap.log


WARNING: ThreadSanitizer: data race (pid=3612)
--
Read of size 8 at 0x7b7000090388 by thread T134:
#0 std::__1::unique_ptr<rocksdb::DB, std::__1::default_delete<rocksdb::DB> >::operator bool() const $YB_THIRDPARTY_DIR/installed/tsan/libcxx/include/c++/v1/memory:2619:19 (libtablet.so+0x13daea)
#1 yb::tablet::Tablet::Flush(yb::tablet::FlushMode, yb::tablet::FlushFlags, long) /net/centos-gcp-cloud-jenkins-slave-ba0/share/jenkins/workspace/github-yugabyte-db-phabricator/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/tablet.cc:1358 (libtablet.so+0x13daea)
...

Previous write of size 8 at 0x7b7000090388 by thread T109 (mutexes: write M1824570827407696):
-- | --
#0 std::__1::unique_ptr<rocksdb::DB, std::__1::default_delete<rocksdb::DB> >::reset(rocksdb::DB*) $YB_THIRDPARTY_DIR/installed/tsan/libcxx/include/c++/v1/memory:2632:20 (libtablet.so+0x135b29)
#1 yb::tablet::Tablet::Shutdown() /net/centos-gcp-cloud-jenkins-slave-ba0/share/jenkins/workspace/github-yugabyte-db-phabricator/build/tsan-clang-dynamic-ninja/../../src/yb/tablet/tablet.cc:667 (libtablet.so+0x135b29)
...
@bmatican bmatican added kind/bug This issue is a bug area/docdb YugabyteDB core features labels Aug 19, 2019
@bmatican bmatican changed the title [docdb] Race between Tablet::Flush and Tablet::Shutdown [docdb] Tsan race on QLStressTest_LongRemoteBootstrap Dec 6, 2019
@ttyusupov ttyusupov assigned ttyusupov and unassigned spolitov Oct 2, 2020
ttyusupov added a commit that referenced this issue Oct 5, 2020
@ttyusupov
#2101: Fixed race between Tablet::Flush and Tablet::Shutdown
934feb2 
Summary:
```
WARNING: ThreadSanitizer: data race (pid=11311)
1762	   Read of size 8 at 0x7b74000cfb58 by thread T155:
1763	     #0 std::__1::unique_ptr<rocksdb::DB, std::__1::default_delete<rocksdb::DB> >::operator bool() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20200829090443-f431681041-centos/installed/tsan/libcxx/include/c++/v1/memory:2619:19 (libtablet.so+0x211067)
1764	     #1 yb::tablet::Tablet::Flush(yb::tablet::FlushMode, yb::tablet::FlushFlags, long) src/yb/tablet/tablet.cc:1847 (libtablet.so+0x211067)
1765	     #2 yb::tserver::MiniTabletServer::FlushTablets(yb::tablet::FlushMode, yb::tablet::FlushFlags)::$_2::operator()(yb::tablet::TabletPeer*) const src/yb/tserver/mini_tablet_server.cc:200:35 (libtserver.so+0x1a0997)
...
1771	     #8 yb::tserver::MiniTabletServer::FlushTablets(yb::tablet::FlushMode, yb::tablet::FlushFlags) src/yb/tserver/mini_tablet_server.cc:196:10 (libtserver.so+0x19f4e8)
1772	     #9 yb::MiniCluster::FlushTablets(yb::tablet::FlushMode, yb::tablet::FlushFlags) src/yb/integration-tests/mini_cluster.cc:369:5 (libintegration-tests.so+0x10cf88)
1773	     #10 yb::client::QLStressTest_LongRemoteBootstrap_Test::TestBody()::$_8::operator()() const src/yb/client/ql-stress-test.cc:972:7 (ql-stress-test+0x4f2f1d)

Previous write of size 8 at 0x7b74000cfb58 by thread T49 (mutexes: write M263877791325157728):
1780	     #0 std::__1::unique_ptr<rocksdb::DB, std::__1::default_delete<rocksdb::DB> >::reset(rocksdb::DB*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20200829090443-f431681041-centos/installed/tsan/libcxx/include/c++/v1/memory:2632:20 (libtablet.so+0x209c9c)
1781	     #1 yb::tablet::ResetRocksDB(bool, rocksdb::Options const&, std::__1::unique_ptr<rocksdb::DB, std::__1::default_delete<rocksdb::DB> >*) src/yb/tablet/tablet.cc:950 (libtablet.so+0x209c9c)
1782	     #2 yb::tablet::Tablet::ResetRocksDBs(yb::StronglyTypedBool<yb::tablet::Destroy_Tag>, yb::StronglyTypedBool<yb::tablet::DisableFlushOnShutdown_Tag>) src/yb/tablet/tablet.cc:967:27 (libtablet.so+0x209b3b)
1783	     #3 yb::tablet::Tablet::CompleteShutdown(yb::StronglyTypedBool<yb::tablet::IsDropTable_Tag>) src/yb/tablet/tablet.cc:931:3 (libtablet.so+0x205bac)
1784	     #4 yb::tablet::TabletPeer::CompleteShutdown(yb::StronglyTypedBool<yb::tablet::IsDropTable_Tag>) src/yb/tablet/tablet_peer.cc:475:14 (libtablet.so+0x287f71)
1785	     #5 yb::tablet::TabletPeer::Shutdown(yb::StronglyTypedBool<yb::tablet::IsDropTable_Tag>) src/yb/tablet/tablet_peer.cc:529:5 (libtablet.so+0x28896d)
1786	     #6 yb::tserver::TSTabletManager::DeleteTablet(string const&, yb::tablet::TabletDataState, boost::optional<long> const&, boost::optional<yb::tserver::TabletServerErrorPB_Code>*) src/yb/tserver/ts_tablet_manager.cc:1297:16 (libtserver.so+0x22d59e)
1787	     #7 yb::tserver::TabletServiceAdminImpl::DeleteTablet(yb::tserver::DeleteTabletRequestPB const*, yb::tserver::DeleteTabletResponsePB*, yb::rpc::RpcContext) src/yb/tserver/tablet_service.cc:1158:41 (libtserver.so+0x1e1de7)
1788	     #8 yb::tserver::TabletServerAdminServiceIf::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) src/yb/tserver/tserver_admin.service.cc:130:7 (libtserver_admin_proto.so+0x73cb0)
1789	     #9 yb::rpc::ServicePoolImpl::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) src/yb/rpc/service_pool.cc:262:19 (libyrpc.so+0x20aa57)
1790	     #10 yb::rpc::InboundCall::InboundCallTask::Run() src/yb/rpc/inbound_call.cc:212:13 (libyrpc.so+0x1745ee)
```

Added missing `ScopedRWOperation` into `Tablet::Flush` to avoid destroying tablets during flush.

Test Plan: `ybd --remote --dltp tsan --cxx-test client_ql-stress-test --gtest_filter QLStressTest.LongRemoteBootstrap -n 500 -- -p 1`

Reviewers: bogdan, mikhail

Reviewed By: mikhail

Subscribers: zyu, ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D9541
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ttyusupov ttyusupov

Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bmatican @spolitov @ttyusupov