Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Concurrency between data sharing and segment compaction may leave some DMFileReader can not be removed #5956

Closed
JaySon-Huang opened this issue Sep 20, 2022 · 2 comments · Fixed by #5958
Closed

Concurrency between data sharing and segment compaction may leave some DMFileReader can not be removed #5956

JaySon-Huang opened this issue Sep 20, 2022 · 2 comments · Fixed by #5958
Assignees
@JinheLin
Labels
affects-6.3 component/storage severity/critical type/bug The issue is confirmed as a bug.

Comments

@JaySon-Huang
Copy link
Contributor

JaySon-Huang commented Sep 20, 2022 •
edited
Loading

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

run tsan

2. What did you expect to see? (Required)

no error detect

3. What did you see instead (Required)

[2022-09-19T18:35:48.082Z]    13292 ms: /tiflash/gtests_dbms DMMinMaxIndexTest.Equal
[2022-09-19T18:35:48.082Z]    49507 ms: /tiflash/gtests_dbms DMMinMaxIndexTest.Or

https://ci.pingcap.net/blue/organizations/jenkins/tiflash-sanitizer-daily/detail/tiflash-sanitizer-daily/534/pipeline/

[2022-09-19T18:30:28.141Z] WARNING: ThreadSanitizer: data race (pid=18034)
[2022-09-19T18:30:28.141Z]   Write of size 4 at 0x7b4800001d4c by main thread (mutexes: write M107940379348698080):
[2022-09-19T18:30:28.141Z]     #0 DB::DM::DMFile::setStatus(DB::DM::DMFile::Status) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFile.h:387:45 (gtests_dbms+0xbb0a950)
[2022-09-19T18:30:28.141Z]     #1 DB::DM::DMFile::remove(std::__1::shared_ptr<DB::FileProvider> const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFile.cpp:803:13 (gtests_dbms+0xbb0a950)
[2022-09-19T18:30:28.141Z]     #2 DB::DM::StableValueSpace::drop(std::__1::shared_ptr<DB::FileProvider> const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/StableValueSpace.cpp:320:15 (gtests_dbms+0xba9b672)
[2022-09-19T18:30:28.141Z]     #3 DB::DM::Segment::drop(std::__1::shared_ptr<DB::FileProvider> const&, DB::DM::WriteBatches&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:1621:13 (gtests_dbms+0xba68d02)
[2022-09-19T18:30:28.141Z]     #4 DB::DM::DeltaMergeStore::dropAllSegments(bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:360:30 (gtests_dbms+0xb9cec00)
[2022-09-19T18:30:28.141Z]     #5 DB::DM::DeltaMergeStore::drop() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:384:5 (gtests_dbms+0xb9cf6c9)
[2022-09-19T18:30:28.141Z]     #6 DB::DM::tests::checkMatch(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >, std::__1::allocator<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > > >, std::__1::shared_ptr<DB::DM::RSOperator> const&, bool, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_minmax_index.cpp:130:12 (gtests_dbms+0x3e32318)
[2022-09-19T18:30:28.141Z]     #7 DB::DM::tests::DMMinMaxIndexTest_Equal_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_minmax_index.cpp:1055:17 (gtests_dbms+0x3e46553)
[2022-09-19T18:30:28.141Z]     #8 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0x9e1d435)
[2022-09-19T18:30:28.141Z]     #9 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0x9e1d435)
[2022-09-19T18:30:28.141Z]     #10 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0x9df6fd5)
[2022-09-19T18:30:28.141Z]     #11 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0x9df8804)
[2022-09-19T18:30:28.141Z]     #12 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0x9df94cd)
[2022-09-19T18:30:28.141Z]     #13 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0x9e09680)
[2022-09-19T18:30:28.141Z]     #14 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0x9e1e8d5)
[2022-09-19T18:30:28.141Z]     #15 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0x9e1e8d5)
[2022-09-19T18:30:28.141Z]     #16 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0x9e08968)
[2022-09-19T18:30:28.141Z]     #17 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x43c8ed7)
[2022-09-19T18:30:28.141Z]     #18 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:78:16 (gtests_dbms+0x43c8ed7)
[2022-09-19T18:30:28.141Z] 
[2022-09-19T18:30:28.141Z]   Previous read of size 4 at 0x7b4800001d4c by thread T71 (mutexes: write M386183668328678544):
[2022-09-19T18:30:28.141Z]     #0 DB::DM::DMFile::path() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFile.cpp:95:50 (gtests_dbms+0xbb01f04)
[2022-09-19T18:30:28.141Z]     #1 DB::DM::DMFileReader::path() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileReader.h:111:24 (gtests_dbms+0xbbad50c)
[2022-09-19T18:30:28.142Z]     #2 DB::DM::DMFileReaderPool::del(DB::DM::DMFileReader&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/ColumnSharingCache.cpp:34:36 (gtests_dbms+0xbbad50c)
[2022-09-19T18:30:28.142Z]     #3 DB::DM::DMFileBlockInputStream::~DMFileBlockInputStream() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileBlockInputStream.h:46:42 (gtests_dbms+0xbb19625)
[2022-09-19T18:30:28.142Z]     #4 std::__1::__shared_ptr_emplace<DB::DM::DMFileBlockInputStream, std::__1::allocator<DB::DM::DMFileBlockInputStream> >::__on_zero_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:315:24 (gtests_dbms+0xbb18f28)
[2022-09-19T18:30:28.142Z]     #5 std::__1::__shared_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:177:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #6 std::__1::__shared_weak_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:219:27 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #7 std::__1::shared_ptr<DB::IBlockInputStream>::~shared_ptr() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:959:19 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #8 std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> >::destroy(std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/__memory/allocator.h:159:15 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #9 void std::__1::allocator_traits<std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::destroy<std::__1::shared_ptr<DB::IBlockInputStream>, void>(std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> >&, std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/__memory/allocator_traits.h:309:13 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #10 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::__destruct_at_end(std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/vector:450:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #11 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::clear() /usr/local/bin/../include/c++/v1/vector:374:29 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #12 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::~__vector_base() /usr/local/bin/../include/c++/v1/vector:487:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #13 std::__1::vector<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::~vector() /usr/local/bin/../include/c++/v1/vector:579:5 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #14 DB::IBlockInputStream::~IBlockInputStream() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/DataStreams/IBlockInputStream.h:111:42 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #15 std::__1::__shared_ptr_emplace<DB::DM::ConcatSkippableBlockInputStream, std::__1::allocator<DB::DM::ConcatSkippableBlockInputStream> >::__on_zero_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:315:24 (gtests_dbms+0xbaabd58)
[2022-09-19T18:30:28.142Z]     #16 std::__1::__shared_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:177:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #17 std::__1::__shared_weak_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:219:27 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #18 std::__1::shared_ptr<DB::IBlockInputStream>::~shared_ptr() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:959:19 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #19 std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> >::destroy(std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/__memory/allocator.h:159:15 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #20 void std::__1::allocator_traits<std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::destroy<std::__1::shared_ptr<DB::IBlockInputStream>, void>(std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> >&, std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/__memory/allocator_traits.h:309:13 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #21 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::__destruct_at_end(std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/vector:450:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #22 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::clear() /usr/local/bin/../include/c++/v1/vector:374:29 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #23 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::~__vector_base() /usr/local/bin/../include/c++/v1/vector:487:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #24 std::__1::vector<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::~vector() /usr/local/bin/../include/c++/v1/vector:579:5 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #25 DB::IBlockInputStream::~IBlockInputStream() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/DataStreams/IBlockInputStream.h:111:42 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.142Z]     #26 DB::DM::DMRowKeyFilterBlockInputStream<true>::~DMRowKeyFilterBlockInputStream() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/RowKeyFilter.h:161:7 (gtests_dbms+0xba811d0)
[2022-09-19T18:30:28.142Z]     #27 std::__1::__shared_ptr_emplace<DB::DM::DMRowKeyFilterBlockInputStream<true>, std::__1::allocator<DB::DM::DMRowKeyFilterBlockInputStream<true> > >::__on_zero_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:315:24 (gtests_dbms+0xba80138)
[2022-09-19T18:30:28.142Z]     #28 std::__1::__shared_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:177:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.143Z]     #29 std::__1::__shared_weak_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:219:27 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.143Z]     #30 std::__1::shared_ptr<DB::IBlockInputStream>::~shared_ptr() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:959:19 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.143Z]     #31 std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> >::destroy(std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/__memory/allocator.h:159:15 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.143Z]     #32 void std::__1::allocator_traits<std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::destroy<std::__1::shared_ptr<DB::IBlockInputStream>, void>(std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> >&, std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/__memory/allocator_traits.h:309:13 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.143Z]     #33 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::__destruct_at_end(std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/vector:450:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.143Z]     #34 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::clear() /usr/local/bin/../include/c++/v1/vector:374:29 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.143Z]     #35 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::~__vector_base() /usr/local/bin/../include/c++/v1/vector:487:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.143Z]     #36 std::__1::vector<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::~vector() /usr/local/bin/../include/c++/v1/vector:579:5 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.143Z]     #37 DB::IBlockInputStream::~IBlockInputStream() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/DataStreams/IBlockInputStream.h:111:42 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:30:28.143Z]     #38 DB::DM::DMVersionFilterBlockInputStream<0>::~DMVersionFilterBlockInputStream() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DMVersionFilterBlockInputStream.h:81:5 (gtests_dbms+0x3fdd3de)
[2022-09-19T18:30:28.143Z]     #39 std::__1::__shared_ptr_emplace<DB::DM::DMVersionFilterBlockInputStream<0>, std::__1::allocator<DB::DM::DMVersionFilterBlockInputStream<0> > >::__on_zero_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:315:24 (gtests_dbms+0x3fdca58)
[2022-09-19T18:30:28.143Z]     #40 std::__1::__shared_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:177:9 (gtests_dbms+0xbbc1331)
[2022-09-19T18:30:28.143Z]     #41 std::__1::__shared_weak_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:219:27 (gtests_dbms+0xbbc1331)
[2022-09-19T18:30:28.143Z]     #42 std::__1::shared_ptr<DB::IBlockInputStream>::~shared_ptr() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:959:19 (gtests_dbms+0xbbc1331)
[2022-09-19T18:30:28.143Z]     #43 std::__1::shared_ptr<DB::IBlockInputStream>::operator=(std::__1::shared_ptr<DB::IBlockInputStream>&&) /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:990:5 (gtests_dbms+0xbbc1331)
[2022-09-19T18:30:28.143Z]     #44 DB::DM::MergedTask::setStreamFinished(unsigned long) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/MergedTask.h:121:29 (gtests_dbms+0xbbc1331)
[2022-09-19T18:30:28.143Z]     #45 DB::DM::MergedTask::readOneBlock() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/MergedTask.cpp:88:13 (gtests_dbms+0xbbbd6bd)
[2022-09-19T18:30:28.143Z]     #46 DB::DM::MergedTask::readBlock() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/MergedTask.cpp:32:12 (gtests_dbms+0xbbbd1f1)
[2022-09-19T18:30:28.143Z]     #47 DB::DM::SegmentReader::readSegments() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/SegmentReader.cpp:101:39 (gtests_dbms+0xbbc8e42)
[2022-09-19T18:30:28.143Z]     #48 DB::DM::SegmentReader::run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/SegmentReader.cpp:152:13 (gtests_dbms+0xbbc8807)
[2022-09-19T18:30:28.143Z]     #49 decltype(*(static_cast<DB::DM::SegmentReader*>(fp0)).*fp()) std::__1::__invoke<void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*, void>(void (DB::DM::SegmentReader::*&&)(), DB::DM::SegmentReader*&&) /usr/local/bin/../include/c++/v1/type_traits:3859:1 (gtests_dbms+0xbbc9df3)
[2022-09-19T18:30:28.143Z]     #50 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*>&, std::__1::__tuple_indices<2ul>) /usr/local/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xbbc9df3)
[2022-09-19T18:30:28.143Z]     #51 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xbbc9df3)
[2022-09-19T18:30:28.143Z] 
[2022-09-19T18:30:28.143Z]   Location is heap block of size 328 at 0x7b4800001c80 allocated by main thread:
[2022-09-19T18:30:28.143Z]     #0 operator new(unsigned long) /llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:64:3 (gtests_dbms+0x2580f58)
[2022-09-19T18:30:28.143Z]     #1 DB::DM::DMFile::create(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, std::__1::optional<DB::DM::DMChecksumConfig>) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFile.cpp:107:26 (gtests_dbms+0xbb02064)
[2022-09-19T18:30:28.144Z]     #2 DB::DM::writeIntoNewDMFile(DB::DM::DMContext&, std::__1::shared_ptr<std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > > const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::DM::DMFileWriter::Flags) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:109:19 (gtests_dbms+0xba4bf82)
[2022-09-19T18:30:28.144Z]     #3 DB::DM::createNewStable(DB::DM::DMContext&, std::__1::shared_ptr<std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > > const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, unsigned long, DB::DM::WriteBatches&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:174:19 (gtests_dbms+0xba4c760)
[2022-09-19T18:30:28.144Z]     #4 DB::DM::Segment::prepareMergeDelta(DB::DM::DMContext&, std::__1::shared_ptr<std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > > const&, std::__1::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::WriteBatches&) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:664:23 (gtests_dbms+0xba53f93)
[2022-09-19T18:30:28.144Z]     #5 DB::DM::DeltaMergeStore::segmentMergeDelta(DB::DM::DMContext&, std::__1::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::MergeDeltaReason, std::__1::shared_ptr<DB::DM::SegmentSnapshot>) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore_InternalSegment.cpp:458:32 (gtests_dbms+0xba19659)
[2022-09-19T18:30:28.144Z]     #6 DB::DM::DeltaMergeStore::mergeDeltaAll(DB::Context const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:748:9 (gtests_dbms+0xb9d5d5d)
[2022-09-19T18:30:28.144Z]     #7 DB::DM::tests::checkMatch(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >, std::__1::allocator<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > > >, std::__1::shared_ptr<DB::DM::RSOperator> const&, bool, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_minmax_index.cpp:125:12 (gtests_dbms+0x3e31cf8)
[2022-09-19T18:30:28.144Z]     #8 DB::DM::tests::DMMinMaxIndexTest_Equal_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_minmax_index.cpp:1055:17 (gtests_dbms+0x3e46553)
[2022-09-19T18:30:28.144Z]     #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0x9e1d435)
[2022-09-19T18:30:28.144Z]     #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0x9e1d435)
[2022-09-19T18:30:28.144Z]     #11 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0x9df6fd5)
[2022-09-19T18:30:28.144Z]     #12 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0x9df8804)
[2022-09-19T18:30:28.144Z]     #13 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0x9df94cd)
[2022-09-19T18:30:28.144Z]     #14 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0x9e09680)
[2022-09-19T18:30:28.144Z]     #15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0x9e1e8d5)
[2022-09-19T18:30:28.144Z]     #16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0x9e1e8d5)
[2022-09-19T18:30:28.144Z]     #17 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0x9e08968)
[2022-09-19T18:30:28.144Z]     #18 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x43c8ed7)
[2022-09-19T18:30:28.144Z]     #19 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:78:16 (gtests_dbms+0x43c8ed7)
[2022-09-19T18:30:28.144Z] 
[2022-09-19T18:30:28.144Z]   Mutex M107940379348698080 is already destroyed.
[2022-09-19T18:30:28.144Z] 
[2022-09-19T18:30:28.144Z]   Mutex M386183668328678544 is already destroyed.
[2022-09-19T18:30:28.144Z] 
[2022-09-19T18:30:28.144Z]   Thread T71 'SegmentReader' (tid=18694, running) created by main thread at:
[2022-09-19T18:30:28.144Z]     #0 pthread_create /llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:977:3 (gtests_dbms+0x250c37d)
[2022-09-19T18:30:28.144Z]     #1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /usr/local/bin/../include/c++/v1/__threading_support:514:10 (gtests_dbms+0xbbc88e5)
[2022-09-19T18:30:28.145Z]     #2 std::__1::thread::thread<void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*, void>(void (DB::DM::SegmentReader::*&&)(), DB::DM::SegmentReader*&&) /usr/local/bin/../include/c++/v1/thread:307:16 (gtests_dbms+0xbbc88e5)
[2022-09-19T18:30:28.145Z]     #3 DB::DM::SegmentReader::SegmentReader(DB::DM::WorkQueue<std::__1::shared_ptr<DB::DM::MergedTask> >&, std::__1::vector<int, std::__1::allocator<int> > const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/SegmentReader.cpp:35:13 (gtests_dbms+0xbbc86fb)
[2022-09-19T18:30:28.145Z]     #4 std::__1::__unique_if<DB::DM::SegmentReader>::__unique_single std::__1::make_unique<DB::DM::SegmentReader, DB::DM::WorkQueue<std::__1::shared_ptr<DB::DM::MergedTask> >&, std::__1::vector<int, std::__1::allocator<int> > const&>(DB::DM::WorkQueue<std::__1::shared_ptr<DB::DM::MergedTask> >&, std::__1::vector<int, std::__1::allocator<int> > const&) /usr/local/bin/../include/c++/v1/__memory/unique_ptr.h:728:32 (gtests_dbms+0xbbc16e6)
[2022-09-19T18:30:28.145Z]     #5 DB::DM::SegmentReaderPool::SegmentReaderPool(int, std::__1::vector<int, std::__1::allocator<int> > const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/SegmentReader.cpp:179:27 (gtests_dbms+0xbbc16e6)
[2022-09-19T18:30:28.145Z]     #6 std::__1::__unique_if<DB::DM::SegmentReaderPool>::__unique_single std::__1::make_unique<DB::DM::SegmentReaderPool, int&, std::__1::vector<int, std::__1::allocator<int> > const&>(int&, std::__1::vector<int, std::__1::allocator<int> > const&) /usr/local/bin/../include/c++/v1/__memory/unique_ptr.h:728:32 (gtests_dbms+0xbbc2348)
[2022-09-19T18:30:28.145Z]     #7 DB::DM::SegmentReaderPoolManager::init(DB::ServerInfo const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/SegmentReader.cpp:218:32 (gtests_dbms+0xbbc2348)
[2022-09-19T18:30:28.145Z]     #8 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:66:50 (gtests_dbms+0x43c8dca)
[2022-09-19T18:30:28.145Z] 
[2022-09-19T18:30:28.145Z] SUMMARY: ThreadSanitizer: data race /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFile.h:387:45 in DB::DM::DMFile::setStatus(DB::DM::DMFile::Status)
[2022-09-19T18:30:28.145Z] ==================

[2022-09-19T18:32:20.508Z] WARNING: ThreadSanitizer: data race (pid=15248)
[2022-09-19T18:32:20.508Z]   Read of size 4 at 0x7b48000018cc by thread T60 (mutexes: write M387591043212231824):
[2022-09-19T18:32:20.508Z]     #0 DB::DM::DMFile::path() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFile.cpp:95:50 (gtests_dbms+0xbb01f04)
[2022-09-19T18:32:20.508Z]     #1 DB::DM::DMFileReader::path() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileReader.h:111:24 (gtests_dbms+0xbbad50c)
[2022-09-19T18:32:20.508Z]     #2 DB::DM::DMFileReaderPool::del(DB::DM::DMFileReader&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/ColumnSharingCache.cpp:34:36 (gtests_dbms+0xbbad50c)
[2022-09-19T18:32:20.508Z]     #3 DB::DM::DMFileBlockInputStream::~DMFileBlockInputStream() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileBlockInputStream.h:46:42 (gtests_dbms+0xbb19625)
[2022-09-19T18:32:20.508Z]     #4 std::__1::__shared_ptr_emplace<DB::DM::DMFileBlockInputStream, std::__1::allocator<DB::DM::DMFileBlockInputStream> >::__on_zero_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:315:24 (gtests_dbms+0xbb18f28)
[2022-09-19T18:32:20.508Z]     #5 std::__1::__shared_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:177:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #6 std::__1::__shared_weak_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:219:27 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #7 std::__1::shared_ptr<DB::IBlockInputStream>::~shared_ptr() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:959:19 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #8 std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> >::destroy(std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/__memory/allocator.h:159:15 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #9 void std::__1::allocator_traits<std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::destroy<std::__1::shared_ptr<DB::IBlockInputStream>, void>(std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> >&, std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/__memory/allocator_traits.h:309:13 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #10 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::__destruct_at_end(std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/vector:450:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #11 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::clear() /usr/local/bin/../include/c++/v1/vector:374:29 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #12 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::~__vector_base() /usr/local/bin/../include/c++/v1/vector:487:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #13 std::__1::vector<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::~vector() /usr/local/bin/../include/c++/v1/vector:579:5 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #14 DB::IBlockInputStream::~IBlockInputStream() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/DataStreams/IBlockInputStream.h:111:42 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #15 std::__1::__shared_ptr_emplace<DB::DM::ConcatSkippableBlockInputStream, std::__1::allocator<DB::DM::ConcatSkippableBlockInputStream> >::__on_zero_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:315:24 (gtests_dbms+0xbaabd58)
[2022-09-19T18:32:20.508Z]     #16 std::__1::__shared_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:177:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #17 std::__1::__shared_weak_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:219:27 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #18 std::__1::shared_ptr<DB::IBlockInputStream>::~shared_ptr() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:959:19 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #19 std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> >::destroy(std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/__memory/allocator.h:159:15 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.508Z]     #20 void std::__1::allocator_traits<std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::destroy<std::__1::shared_ptr<DB::IBlockInputStream>, void>(std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> >&, std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/__memory/allocator_traits.h:309:13 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #21 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::__destruct_at_end(std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/vector:450:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #22 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::clear() /usr/local/bin/../include/c++/v1/vector:374:29 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #23 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::~__vector_base() /usr/local/bin/../include/c++/v1/vector:487:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #24 std::__1::vector<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::~vector() /usr/local/bin/../include/c++/v1/vector:579:5 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #25 DB::IBlockInputStream::~IBlockInputStream() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/DataStreams/IBlockInputStream.h:111:42 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #26 DB::DM::DMRowKeyFilterBlockInputStream<true>::~DMRowKeyFilterBlockInputStream() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/RowKeyFilter.h:161:7 (gtests_dbms+0xba811d0)
[2022-09-19T18:32:20.509Z]     #27 std::__1::__shared_ptr_emplace<DB::DM::DMRowKeyFilterBlockInputStream<true>, std::__1::allocator<DB::DM::DMRowKeyFilterBlockInputStream<true> > >::__on_zero_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:315:24 (gtests_dbms+0xba80138)
[2022-09-19T18:32:20.509Z]     #28 std::__1::__shared_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:177:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #29 std::__1::__shared_weak_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:219:27 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #30 std::__1::shared_ptr<DB::IBlockInputStream>::~shared_ptr() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:959:19 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #31 std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> >::destroy(std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/__memory/allocator.h:159:15 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #32 void std::__1::allocator_traits<std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::destroy<std::__1::shared_ptr<DB::IBlockInputStream>, void>(std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> >&, std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/__memory/allocator_traits.h:309:13 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #33 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::__destruct_at_end(std::__1::shared_ptr<DB::IBlockInputStream>*) /usr/local/bin/../include/c++/v1/vector:450:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #34 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::clear() /usr/local/bin/../include/c++/v1/vector:374:29 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #35 std::__1::__vector_base<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::~__vector_base() /usr/local/bin/../include/c++/v1/vector:487:9 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #36 std::__1::vector<std::__1::shared_ptr<DB::IBlockInputStream>, std::__1::allocator<std::__1::shared_ptr<DB::IBlockInputStream> > >::~vector() /usr/local/bin/../include/c++/v1/vector:579:5 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #37 DB::IBlockInputStream::~IBlockInputStream() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/DataStreams/IBlockInputStream.h:111:42 (gtests_dbms+0x3c6e43c)
[2022-09-19T18:32:20.509Z]     #38 DB::DM::DMVersionFilterBlockInputStream<0>::~DMVersionFilterBlockInputStream() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DMVersionFilterBlockInputStream.h:81:5 (gtests_dbms+0x3fdd3de)
[2022-09-19T18:32:20.509Z]     #39 std::__1::__shared_ptr_emplace<DB::DM::DMVersionFilterBlockInputStream<0>, std::__1::allocator<DB::DM::DMVersionFilterBlockInputStream<0> > >::__on_zero_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:315:24 (gtests_dbms+0x3fdca58)
[2022-09-19T18:32:20.509Z]     #40 std::__1::__shared_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:177:9 (gtests_dbms+0xbbc1331)
[2022-09-19T18:32:20.509Z]     #41 std::__1::__shared_weak_count::__release_shared() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:219:27 (gtests_dbms+0xbbc1331)
[2022-09-19T18:32:20.509Z]     #42 std::__1::shared_ptr<DB::IBlockInputStream>::~shared_ptr() /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:959:19 (gtests_dbms+0xbbc1331)
[2022-09-19T18:32:20.509Z]     #43 std::__1::shared_ptr<DB::IBlockInputStream>::operator=(std::__1::shared_ptr<DB::IBlockInputStream>&&) /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:990:5 (gtests_dbms+0xbbc1331)
[2022-09-19T18:32:20.509Z]     #44 DB::DM::MergedTask::setStreamFinished(unsigned long) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/MergedTask.h:121:29 (gtests_dbms+0xbbc1331)
[2022-09-19T18:32:20.509Z]     #45 DB::DM::MergedTask::readOneBlock() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/MergedTask.cpp:88:13 (gtests_dbms+0xbbbd6bd)
[2022-09-19T18:32:20.509Z]     #46 DB::DM::MergedTask::readBlock() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/MergedTask.cpp:32:12 (gtests_dbms+0xbbbd1f1)
[2022-09-19T18:32:20.509Z]     #47 DB::DM::SegmentReader::readSegments() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/SegmentReader.cpp:101:39 (gtests_dbms+0xbbc8e42)
[2022-09-19T18:32:20.509Z]     #48 DB::DM::SegmentReader::run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/SegmentReader.cpp:152:13 (gtests_dbms+0xbbc8807)
[2022-09-19T18:32:20.509Z]     #49 decltype(*(static_cast<DB::DM::SegmentReader*>(fp0)).*fp()) std::__1::__invoke<void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*, void>(void (DB::DM::SegmentReader::*&&)(), DB::DM::SegmentReader*&&) /usr/local/bin/../include/c++/v1/type_traits:3859:1 (gtests_dbms+0xbbc9df3)
[2022-09-19T18:32:20.509Z]     #50 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*>&, std::__1::__tuple_indices<2ul>) /usr/local/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xbbc9df3)
[2022-09-19T18:32:20.510Z]     #51 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xbbc9df3)
[2022-09-19T18:32:20.510Z] 
[2022-09-19T18:32:20.510Z]   Previous write of size 4 at 0x7b48000018cc by main thread (mutexes: write M706919129788974048):
[2022-09-19T18:32:20.510Z]     #0 DB::DM::DMFile::setStatus(DB::DM::DMFile::Status) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFile.h:387:45 (gtests_dbms+0xbb0a950)
[2022-09-19T18:32:20.510Z]     #1 DB::DM::DMFile::remove(std::__1::shared_ptr<DB::FileProvider> const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFile.cpp:803:13 (gtests_dbms+0xbb0a950)
[2022-09-19T18:32:20.510Z]     #2 DB::DM::StableValueSpace::drop(std::__1::shared_ptr<DB::FileProvider> const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/StableValueSpace.cpp:320:15 (gtests_dbms+0xba9b672)
[2022-09-19T18:32:20.510Z]     #3 DB::DM::Segment::drop(std::__1::shared_ptr<DB::FileProvider> const&, DB::DM::WriteBatches&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:1621:13 (gtests_dbms+0xba68d02)
[2022-09-19T18:32:20.510Z]     #4 DB::DM::DeltaMergeStore::dropAllSegments(bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:360:30 (gtests_dbms+0xb9cec00)
[2022-09-19T18:32:20.510Z]     #5 DB::DM::DeltaMergeStore::drop() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:384:5 (gtests_dbms+0xb9cf6c9)
[2022-09-19T18:32:20.510Z]     #6 DB::DM::tests::checkMatch(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >, std::__1::allocator<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > > >, std::__1::shared_ptr<DB::DM::RSOperator> const&, bool, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_minmax_index.cpp:130:12 (gtests_dbms+0x3e32318)
[2022-09-19T18:32:20.510Z]     #7 DB::DM::tests::DMMinMaxIndexTest_Or_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_minmax_index.cpp:1203:21 (gtests_dbms+0x3e5b22b)
[2022-09-19T18:32:20.510Z]     #8 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0x9e1d435)
[2022-09-19T18:32:20.510Z]     #9 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0x9e1d435)
[2022-09-19T18:32:20.510Z]     #10 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0x9df6fd5)
[2022-09-19T18:32:20.510Z]     #11 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0x9df8804)
[2022-09-19T18:32:20.510Z]     #12 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0x9df94cd)
[2022-09-19T18:32:20.510Z]     #13 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0x9e09680)
[2022-09-19T18:32:20.510Z]     #14 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0x9e1e8d5)
[2022-09-19T18:32:20.510Z]     #15 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0x9e1e8d5)
[2022-09-19T18:32:20.510Z]     #16 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0x9e08968)
[2022-09-19T18:32:20.510Z]     #17 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x43c8ed7)
[2022-09-19T18:32:20.510Z]     #18 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:78:16 (gtests_dbms+0x43c8ed7)
[2022-09-19T18:32:20.510Z] 
[2022-09-19T18:32:20.510Z]   Location is heap block of size 328 at 0x7b4800001800 allocated by main thread:
[2022-09-19T18:32:20.510Z]     #0 operator new(unsigned long) /llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:64:3 (gtests_dbms+0x2580f58)
[2022-09-19T18:32:20.510Z]     #1 DB::DM::DMFile::create(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, std::__1::optional<DB::DM::DMChecksumConfig>) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFile.cpp:107:26 (gtests_dbms+0xbb02064)
[2022-09-19T18:32:20.510Z]     #2 DB::DM::writeIntoNewDMFile(DB::DM::DMContext&, std::__1::shared_ptr<std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > > const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::DM::DMFileWriter::Flags) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:109:19 (gtests_dbms+0xba4bf82)
[2022-09-19T18:32:20.510Z]     #3 DB::DM::createNewStable(DB::DM::DMContext&, std::__1::shared_ptr<std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > > const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, unsigned long, DB::DM::WriteBatches&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:174:19 (gtests_dbms+0xba4c760)
[2022-09-19T18:32:20.510Z]     #4 DB::DM::Segment::prepareMergeDelta(DB::DM::DMContext&, std::__1::shared_ptr<std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > > const&, std::__1::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::WriteBatches&) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:664:23 (gtests_dbms+0xba53f93)
[2022-09-19T18:32:20.511Z]     #5 DB::DM::DeltaMergeStore::segmentMergeDelta(DB::DM::DMContext&, std::__1::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::MergeDeltaReason, std::__1::shared_ptr<DB::DM::SegmentSnapshot>) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore_InternalSegment.cpp:458:32 (gtests_dbms+0xba19659)
[2022-09-19T18:32:20.511Z]     #6 DB::DM::DeltaMergeStore::mergeDeltaAll(DB::Context const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:748:9 (gtests_dbms+0xb9d5d5d)
[2022-09-19T18:32:20.511Z]     #7 DB::DM::tests::checkMatch(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >, std::__1::allocator<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > > >, std::__1::shared_ptr<DB::DM::RSOperator> const&, bool, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_minmax_index.cpp:125:12 (gtests_dbms+0x3e31cf8)
[2022-09-19T18:32:20.511Z]     #8 DB::DM::tests::DMMinMaxIndexTest_Or_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_minmax_index.cpp:1203:21 (gtests_dbms+0x3e5b22b)
[2022-09-19T18:32:20.511Z]     #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0x9e1d435)
[2022-09-19T18:32:20.511Z]     #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0x9e1d435)
[2022-09-19T18:32:20.511Z]     #11 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0x9df6fd5)
[2022-09-19T18:32:20.511Z]     #12 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0x9df8804)
[2022-09-19T18:32:20.511Z]     #13 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0x9df94cd)
[2022-09-19T18:32:20.511Z]     #14 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0x9e09680)
[2022-09-19T18:32:20.511Z]     #15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0x9e1e8d5)
[2022-09-19T18:32:20.511Z]     #16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0x9e1e8d5)
[2022-09-19T18:32:20.511Z]     #17 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0x9e08968)
[2022-09-19T18:32:20.511Z]     #18 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x43c8ed7)
[2022-09-19T18:32:20.511Z]     #19 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:78:16 (gtests_dbms+0x43c8ed7)
[2022-09-19T18:32:20.511Z] 
[2022-09-19T18:32:20.511Z]   Mutex M387591043212231824 is already destroyed.
[2022-09-19T18:32:20.511Z] 
[2022-09-19T18:32:20.511Z]   Mutex M706919129788974048 is already destroyed.
[2022-09-19T18:32:20.511Z] 
[2022-09-19T18:32:20.511Z]   Thread T60 'SegmentReader' (tid=15642, running) created by main thread at:
[2022-09-19T18:32:20.511Z]     #0 pthread_create /llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:977:3 (gtests_dbms+0x250c37d)
[2022-09-19T18:32:20.511Z]     #1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /usr/local/bin/../include/c++/v1/__threading_support:514:10 (gtests_dbms+0xbbc88e5)
[2022-09-19T18:32:20.511Z]     #2 std::__1::thread::thread<void (DB::DM::SegmentReader::*)(), DB::DM::SegmentReader*, void>(void (DB::DM::SegmentReader::*&&)(), DB::DM::SegmentReader*&&) /usr/local/bin/../include/c++/v1/thread:307:16 (gtests_dbms+0xbbc88e5)
[2022-09-19T18:32:20.511Z]     #3 DB::DM::SegmentReader::SegmentReader(DB::DM::WorkQueue<std::__1::shared_ptr<DB::DM::MergedTask> >&, std::__1::vector<int, std::__1::allocator<int> > const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/SegmentReader.cpp:35:13 (gtests_dbms+0xbbc86fb)
[2022-09-19T18:32:20.511Z]     #4 std::__1::__unique_if<DB::DM::SegmentReader>::__unique_single std::__1::make_unique<DB::DM::SegmentReader, DB::DM::WorkQueue<std::__1::shared_ptr<DB::DM::MergedTask> >&, std::__1::vector<int, std::__1::allocator<int> > const&>(DB::DM::WorkQueue<std::__1::shared_ptr<DB::DM::MergedTask> >&, std::__1::vector<int, std::__1::allocator<int> > const&) /usr/local/bin/../include/c++/v1/__memory/unique_ptr.h:728:32 (gtests_dbms+0xbbc16e6)
[2022-09-19T18:32:20.511Z]     #5 DB::DM::SegmentReaderPool::SegmentReaderPool(int, std::__1::vector<int, std::__1::allocator<int> > const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/SegmentReader.cpp:179:27 (gtests_dbms+0xbbc16e6)
[2022-09-19T18:32:20.511Z]     #6 std::__1::__unique_if<DB::DM::SegmentReaderPool>::__unique_single std::__1::make_unique<DB::DM::SegmentReaderPool, int&, std::__1::vector<int, std::__1::allocator<int> > const&>(int&, std::__1::vector<int, std::__1::allocator<int> > const&) /usr/local/bin/../include/c++/v1/__memory/unique_ptr.h:728:32 (gtests_dbms+0xbbc2348)
[2022-09-19T18:32:20.511Z]     #7 DB::DM::SegmentReaderPoolManager::init(DB::ServerInfo const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/ReadThread/SegmentReader.cpp:218:32 (gtests_dbms+0xbbc2348)
[2022-09-19T18:32:20.511Z]     #8 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:66:50 (gtests_dbms+0x43c8dca)
[2022-09-19T18:32:20.511Z] 
[2022-09-19T18:32:20.512Z] SUMMARY: ThreadSanitizer: data race /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/File/DMFile.cpp:95:50 in DB::DM::DMFile::path() const

4. What is your TiFlash version? (Required)
@JaySon-Huang JaySon-Huang added the type/bug The issue is confirmed as a bug. label Sep 20, 2022
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Sep 20, 2022 •
edited
Loading

Thread A Thread B
create inputstream and push a reader with path "/xx/dmf_100" to DMFileReaderPool
Segment Merge/MergeDelta and rewrite the data from dmf_100 to another file
read done, release snapshot, but the reader is not yet released
the dmf_100 is not ref by any available segments and reading snapshot, change the status to DROPPED and remove the data from disk
release reader, try to get the path of dmfile, but as the file status is DROPPED, can not release the DMFileReader * from memory

/cc @JinheLin

@JaySon-Huang
Copy link
Contributor Author

I think we can resolve this issue by:

  • Add a RUNTIME_CHECK for ensuring that when creating DMFileReader, the related DMFile status must be READABLE
  • No matter what is the current status of DMFile, always use DMFile::getPathByStatus with DMFile::Status::READABLE when releasing dmfileReader from the reader pool

@JinheLin JinheLin mentioned this issue Sep 20, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Sep 21, 2022
@JinheLin
Always use readable pathname in DMFileReader (#5958)
7c0472c 
close #5956
@ti-chi-bot ti-chi-bot mentioned this issue Sep 21, 2022
Merged
12 tasks
ti-chi-bot added a commit that referenced this issue Sep 22, 2022
@ti-chi-bot
Always use readable pathname in DMFileReader (#5958) (#5983)
6a305d3 
close #5956
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JinheLin JinheLin

Labels
affects-6.3 component/storage severity/critical type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Always use readable pathname in DMFileReader JinheLin/tiflash-1
4 participants
@flowbehappy @JaySon-Huang @JinheLin @ti-chi-bot