Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ASAN/TSan failed compute test case #8285

Open
yibin87 opened this issue Nov 1, 2023 · 4 comments
Open

ASAN/TSan failed compute test case #8285

yibin87 opened this issue Nov 1, 2023 · 4 comments
Labels
component/compute priority/P2 The issue has P2 priority. severity/minor type/bug The issue is confirmed as a bug.

Comments

@yibin87
Copy link
Contributor

yibin87 commented Nov 1, 2023

Use 2023-10-31 latest version, error message:
[2023-10-31T18:37:38.841Z] WARNING: ThreadSanitizer: data race (pid=20500)
[2023-10-31T18:37:38.841Z] Read of size 8 at 0x7b6000030208 by main thread:
[2023-10-31T18:37:38.841Z] #0 memcpy /llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:827:5 (gtests_dbms+0x2ba6ce0)
[2023-10-31T18:37:38.841Z] #1 core::mem::maybe_uninit::MaybeUninit$LT$T$GT$::assume_init::h2f3b2e6915ec01bd /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/mem/maybe_uninit.rs:627:38 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #2 core::ptr::read::h5d271dd8cc4a87b2 /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/ptr/mod.rs:1140:9 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #3 alloc::vec::Vec$LT$T$C$A$GT$::remove::h77cb7968e7379f6d /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/vec/mod.rs:1522:23 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #4 backtrace::symbolize::gimli::Cache::mapping_for_lib::h8e9c09979b3359f8 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:312:29 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #5 backtrace::symbolize::gimli::resolve::_$u7b$$u7b$closure$u7d$$u7d$::hc070d62f1c0573e8 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:354:24 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #6 backtrace::symbolize::gimli::Cache::with_global::h9110e7656f265813 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:266:9 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #7 backtrace::symbolize::gimli::resolve::h56abb7b895e03eed /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:346:5 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #8 DB::tests::ComputeServerRunner_aggWithColumnPrune_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/tests/gtest_compute_server.cpp:564:1 (gtests_dbms+0x312650a)
[2023-10-31T18:37:38.841Z] #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc2bd085)
[2023-10-31T18:37:38.841Z] #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc2bd085)
[2023-10-31T18:37:38.841Z] #11 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xc296c25)
[2023-10-31T18:37:38.841Z] #12 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xc298454)
[2023-10-31T18:37:38.841Z] #13 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xc29911d)
[2023-10-31T18:37:38.841Z] #14 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xc2a92d0)
[2023-10-31T18:37:38.841Z] #15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc2be525)
[2023-10-31T18:37:38.841Z] #16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc2be525)
[2023-10-31T18:37:38.841Z] #17 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xc2a85b8)
[2023-10-31T18:37:38.841Z] #18 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x5f0b9ee)
[2023-10-31T18:37:38.841Z] #19 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x5f0b9ee)
[2023-10-31T18:37:38.841Z]
[2023-10-31T18:37:38.841Z] Previous write of size 8 at 0x7b6000030208 by thread T1456:
[2023-10-31T18:37:38.841Z] [failed to restore the stack]
[2023-10-31T18:37:38.841Z]
[2023-10-31T18:37:38.841Z] Location is heap block of size 1024 at 0x7b6000030000 allocated by thread T1450:
[2023-10-31T18:37:38.841Z] #0 malloc /llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:655:5 (gtests_dbms+0x2b9ba2d)
[2023-10-31T18:37:38.841Z] #1 alloc::alloc::alloc::h9f5d2080c32a5558 /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/alloc.rs:95:14 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #2 alloc::alloc::Global::alloc_impl::h2b2278f322e264e8 /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/alloc.rs:177:73 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #3 $LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$::allocate::h22ac9faea77a7755 /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/alloc.rs:237:9 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #4 alloc::raw_vec::RawVec$LT$T$C$A$GT$::allocate_in::hd28f1e60fbe07f7c /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/raw_vec.rs:185:45 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #5 alloc::raw_vec::RawVec$LT$T$C$A$GT$::with_capacity_in::h73c2188b6b63f3ac /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/raw_vec.rs:131:9 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #6 alloc::vec::Vec$LT$T$C$A$GT$::with_capacity_in::hfdce392da93ae143 /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/vec/mod.rs:673:20 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #7 alloc::vec::Vec$LT$T$GT$::with_capacity::hb875a34790c246fd /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/vec/mod.rs:483:9 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #8 backtrace::symbolize::gimli::Cache::new::h0b35aa2a5cc50baa /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:247:23 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #9 backtrace::symbolize::gimli::Cache::with_global::$u7b$$u7b$closure$u7d$$u7d$::h917b603032c4e945 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:266:48 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #10 core::option::Option$LT$T$GT$::get_or_insert_with::h0869f117ab9c441a /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/option.rs:1546:49 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #11 backtrace::symbolize::gimli::Cache::with_global::h9110e7656f265813 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:266:11 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #12 backtrace::symbolize::gimli::resolve::h56abb7b895e03eed /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:346:5 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #13 DB::getExceptionMessage(DB::Exception const&, bool, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/Exception.cpp:217:74 (gtests_dbms+0xf8c1f5a)
[2023-10-31T18:37:38.841Z] #14 DB::getCurrentExceptionMessage(bool, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/Exception.cpp:106:23 (gtests_dbms+0xf8c0962)
[2023-10-31T18:37:38.841Z] #15 DB::MPPTask::runImpl() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Mpp/MPPTask.cpp:602:30 (gtests_dbms+0xf054ec0)
[2023-10-31T18:37:38.841Z] #16 DB::MPPTask::run()::$_19::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Mpp/MPPTask.cpp:203:97 (gtests_dbms+0xf056f63)
[2023-10-31T18:37:38.841Z] #17 decltype(static_castDB::MPPTask::run()::$_19&(fp)()) std::__1::__invokeDB::MPPTask::run()::$_19&(DB::MPPTask::run()::$_19&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xf056f63)
[2023-10-31T18:37:38.842Z] #18 void std::__1::__invoke_void_return_wrapper<void, true>::__callDB::MPPTask::run()::$_19&(DB::MPPTask::run()::$_19&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0xf056f63)
[2023-10-31T18:37:38.842Z] #19 std::__1::__function::__alloc_func<DB::MPPTask::run()::$_19, std::__1::allocatorDB::MPPTask::run()::$_19, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0xf056f63)
[2023-10-31T18:37:38.842Z] #20 std::__1::__function::__func<DB::MPPTask::run()::$_19, std::__1::allocatorDB::MPPTask::run()::$_19, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0xf056f63)
[2023-10-31T18:37:38.842Z] #21 std::__1::__function::__value_func<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xf91ec5d)
[2023-10-31T18:37:38.842Z] #22 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xf91ec5d)
[2023-10-31T18:37:38.842Z] #23 decltype(static_cast<std::__1::function<void ()> const&>(fp)()) std::__1::__invoke<std::__1::function<void ()> const&>(std::__1::function<void ()> const&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xf91ec5d)
[2023-10-31T18:37:38.842Z] #24 std::__1::invoke_result<std::__1::function<void ()> const&>::type std::__1::invoke<std::__1::function<void ()> const&>(std::__1::function<void ()> const&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:93:12 (gtests_dbms+0xf91ec5d)
[2023-10-31T18:37:38.842Z] #25 auto std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::function<void ()>&&)::'lambda'(auto&&...)::operator()<>(auto&&...) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/ThreadFactory.h:48:26 (gtests_dbms+0xf91ec5d)
[2023-10-31T18:37:38.842Z] #26 decltype(static_cast<std::__1::function<void ()>>(fp)()) std::__1::__invoke<std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::function<void ()>&&)::'lambda'(auto&&...)>(std::__1::function<void ()>&&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xf91e8e7)
[2023-10-31T18:37:38.842Z] #27 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::function<void ()>&&)::'lambda'(auto&&...)>(std::__1::tuple<std::__1::function<void ()>, std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::function<void ()>&&)::'lambda'(auto&&...)>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xf91e8e7)
[2023-10-31T18:37:38.842Z] #28 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::function<void ()>&&)::'lambda'(auto&&...)> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xf91e8e7)
@yibin87 yibin87 added type/bug The issue is confirmed as a bug. priority/P2 The issue has P2 priority. severity/minor component/compute labels Nov 1, 2023
@yibin87
Copy link
Contributor Author

yibin87 commented Dec 12, 2023 •
edited
Loading

False positive cases containing google::protobuf::internal::AssignDescriptors:
[2023-12-11T18:31:39.584Z] ==================
[2023-12-11T18:31:39.584Z] WARNING: ThreadSanitizer: data race (pid=5177)
[2023-12-11T18:31:39.584Z] Read of size 8 at 0x0000149dea60 by thread T972:
[2023-12-11T18:31:39.584Z] #0 google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* ()(), std::__1::once_flag, google::protobuf::Metadata const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/generated_message_reflection.cc:3007:10 (gtests_dbms+0x10def781)
[2023-12-11T18:31:39.584Z] #1 mpp::EstablishMPPConnectionRequest::GetMetadata() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/build-TSan/contrib/kvproto/cpp/kvproto/mpp.pb.cc:3058:10 (gtests_dbms+0x102ecef3)
[2023-12-11T18:31:39.584Z] #2 google::protobuf::Message::GetReflection() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/message.h:358:52 (gtests_dbms+0x10e1befa)
[2023-12-11T18:31:39.584Z] #3 google::protobuf::TextFormat::Printer::Print(google::protobuf::Message const&, google::protobuf::TextFormat::Printer::TextGenerator*) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/text_format.cc:2165:42 (gtests_dbms+0x10e1befa)
[2023-12-11T18:31:39.584Z] #4 google::protobuf::TextFormat::Printer::Print(google::protobuf::Message const&, google::protobuf::io::ZeroCopyOutputStream*) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/text_format.cc:2074:3 (gtests_dbms+0x10e178bf)
[2023-12-11T18:31:39.584Z] #5 google::protobuf::TextFormat::Printer::PrintToString(google::protobuf::Message const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/text_format.cc:2058:10 (gtests_dbms+0x10e178bf)
[2023-12-11T18:31:39.584Z] #6 google::protobuf::Message::DebugString() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/text_format.cc:101:11 (gtests_dbms+0x10e176ca)
[2023-12-11T18:31:39.584Z] #7 DB::AsyncFlashService::establishMPPConnectionAsync(DB::EstablishCallData) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/FlashService.cpp:599:5 (gtests_dbms+0xef40cb4)
[2023-12-11T18:31:39.584Z] #8 DB::EstablishCallData::initRpc() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/EstablishCall.cpp:170:29 (gtests_dbms+0xef2dff5)
[2023-12-11T18:31:39.584Z] #9 DB::EstablishCallData::execute(bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/EstablishCall.cpp:85:9 (gtests_dbms+0xef2dc1e)
[2023-12-11T18:31:39.584Z] #10 non-virtual thunk to DB::EstablishCallData::execute(bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/EstablishCall.cpp (gtests_dbms+0xef300b4)
[2023-12-11T18:31:39.584Z] #11 DB::(anonymous namespace)::handleRpcs(grpc::ServerCompletionQueue*, std::__1::shared_ptrDB::Logger const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Server/FlashGrpcServerHolder.cpp:62:51 (gtests_dbms+0xd3c64a9)
[2023-12-11T18:31:39.584Z] #12 DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Server/FlashGrpcServerHolder.cpp:206:85 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.584Z] #13 decltype(static_cast<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8 const&>(fp)()) std::__1::__invoke<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8 const&>(DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8 const&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.584Z] #14 std::__1::invoke_result<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8 const&>::type std::__1::invoke<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8 const&>(DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8 const&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:93:12 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.584Z] #15 auto std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&)::'lambda'(auto&&...)::operator()<>(auto&&...) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/ThreadFactory.h:48:26 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.584Z] #16 decltype(static_cast<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(fp)()) std::__1::__invoke<std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&)::'lambda'(auto&&...)>(DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.585Z] #17 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&)::'lambda'(auto&&...)>(std::__1::tuple<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8, std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&)::'lambda'(auto&&...)>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.585Z] #18 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&)::'lambda'(auto&&...)> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.585Z]
[2023-12-11T18:31:39.585Z] Previous write of size 8 at 0x0000149dea60 by thread T1346:
[2023-12-11T18:31:39.585Z] #0 google::protobuf::AssignDescriptorsHelper::AssignMessageDescriptor(google::protobuf::Descriptor const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/generated_message_reflection.cc:2835:38 (gtests_dbms+0x10df9bfd)
[2023-12-11T18:31:39.585Z] #1 google::protobuf::(anonymous namespace)::AssignDescriptorsImpl(google::protobuf::internal::DescriptorTable const*, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/generated_message_reflection.cc:2942:12 (gtests_dbms+0x10defb85)
[2023-12-11T18:31:39.585Z] #2 google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* ()(), std::__1::once_flag, google::protobuf::Metadata const&)::$_0::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/generated_message_reflection.cc:3004:5 (gtests_dbms+0x10df1dcc)
[2023-12-11T18:31:39.585Z] #3 decltype(static_cast<google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* ()(), std::__1::once_flag, google::protobuf::Metadata const&)::$_0>(fp)()) std::__1::__invoke<google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* ()(), std::__1::once_flag, google::protobuf::Metadata const&)::$_0>(google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* ()(), std::__1::once_flag, google::protobuf::Metadata const&)::$_0&&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0x10df1dcc)
[2023-12-11T18:31:39.585Z] #4 void std::__1::__call_once_param<std::__1::tuple<google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* ()(), std::__1::once_flag, google::protobuf::Metadata const&)::$_0&&> >::__execute<>(std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/mutex:630:9 (gtests_dbms+0x10df1dcc)
[2023-12-11T18:31:39.585Z] #5 std::__1::__call_once_param<std::__1::tuple<google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* ()(), std::__1::once_flag, google::protobuf::Metadata const&)::$_0&&> >::operator()() /usr/local/bin/../include/c++/v1/mutex:622:9 (gtests_dbms+0x10df1dcc)
[2023-12-11T18:31:39.585Z] #6 void std::__1::__call_once_proxy<std::__1::tuple<google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* ()(), std::__1::once_flag, google::protobuf::Metadata const&)::$_0&&> >(void*) /usr/local/bin/../include/c++/v1/mutex:658:5 (gtests_dbms+0x10df1dcc)
[2023-12-11T18:31:39.585Z] #7 std::__1::__call_once(unsigned long volatile&, void*, void ()(void)) (libc++.so.1+0x67487)
[2023-12-11T18:31:39.585Z] #8 mpp::EstablishMPPConnectionRequest::GetMetadata() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/build-TSan/contrib/kvproto/cpp/kvproto/mpp.pb.cc:3058:10 (gtests_dbms+0x102ecef3)
[2023-12-11T18:31:39.585Z] #9 google::protobuf::Message::GetReflection() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/message.h:358:52 (gtests_dbms+0x10e1befa)
[2023-12-11T18:31:39.585Z] #10 google::protobuf::TextFormat::Printer::Print(google::protobuf::Message const&, google::protobuf::TextFormat::Printer::TextGenerator*) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/text_format.cc:2165:42 (gtests_dbms+0x10e1befa)
[2023-12-11T18:31:39.585Z] #11 google::protobuf::TextFormat::Printer::Print(google::protobuf::Message const&, google::protobuf::io::ZeroCopyOutputStream*) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/text_format.cc:2074:3 (gtests_dbms+0x10e178bf)
[2023-12-11T18:31:39.585Z] #12 google::protobuf::TextFormat::Printer::PrintToString(google::protobuf::Message const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/text_format.cc:2058:10 (gtests_dbms+0x10e178bf)
[2023-12-11T18:31:39.585Z] #13 google::protobuf::Message::DebugString() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/text_format.cc:101:11 (gtests_dbms+0x10e176ca)
[2023-12-11T18:31:39.585Z] #14 DB::AsyncFlashService::establishMPPConnectionAsync(DB::EstablishCallData) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/FlashService.cpp:599:5 (gtests_dbms+0xef40cb4)
[2023-12-11T18:31:39.585Z] #15 DB::EstablishCallData::initRpc() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/EstablishCall.cpp:170:29 (gtests_dbms+0xef2dff5)
[2023-12-11T18:31:39.585Z] #16 DB::EstablishCallData::execute(bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/EstablishCall.cpp:85:9 (gtests_dbms+0xef2dc1e)
[2023-12-11T18:31:39.585Z] #17 non-virtual thunk to DB::EstablishCallData::execute(bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/EstablishCall.cpp (gtests_dbms+0xef300b4)
[2023-12-11T18:31:39.585Z] #18 DB::(anonymous namespace)::handleRpcs(grpc::ServerCompletionQueue*, std::__1::shared_ptrDB::Logger const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Server/FlashGrpcServerHolder.cpp:62:51 (gtests_dbms+0xd3c64a9)
[2023-12-11T18:31:39.585Z] #19 DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Server/FlashGrpcServerHolder.cpp:206:85 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.585Z] #20 decltype(static_cast<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8 const&>(fp)()) std::__1::__invoke<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8 const&>(DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8 const&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.585Z] #21 std::__1::invoke_result<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8 const&>::type std::__1::invoke<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8 const&>(DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8 const&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:93:12 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.585Z] #22 auto std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&)::'lambda'(auto&&...)::operator()<>(auto&&...) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/ThreadFactory.h:48:26 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.586Z] #23 decltype(static_cast<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(fp)()) std::__1::__invoke<std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&)::'lambda'(auto&&...)>(DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.586Z] #24 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&)::'lambda'(auto&&...)>(std::__1::tuple<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8, std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&)::'lambda'(auto&&...)>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.586Z] #25 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&)::'lambda'(auto&&...)> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xd3c7218)
[2023-12-11T18:31:39.586Z]
[2023-12-11T18:31:39.586Z] Location is global 'file_level_metadata_mpp_2eproto' of size 192 at 0x0000149de9d0 (gtests_dbms+0x0000149dea60)
[2023-12-11T18:31:39.586Z]
[2023-12-11T18:31:39.586Z] Thread T972 'async_poller' (tid=7154, running) created by main thread at:
[2023-12-11T18:31:39.586Z] #0 pthread_create /llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:977:3 (gtests_dbms+0x2b8717d)
[2023-12-11T18:31:39.586Z] #1 std::__1::__libcpp_thread_create(unsigned long*, void* ()(void), void*) /usr/local/bin/../include/c++/v1/__threading_support:514:10 (gtests_dbms+0xd3c45cf)
[2023-12-11T18:31:39.586Z] #2 std::__1::thread::thread<std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&)::'lambda'(auto&&...)&, void>(DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&) /usr/local/bin/../include/c++/v1/thread:307:16 (gtests_dbms+0xd3c45cf)
[2023-12-11T18:31:39.586Z] #3 std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/ThreadFactory.h:50:16 (gtests_dbms+0xd3c45cf)
[2023-12-11T18:31:39.586Z] #4 DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Server/FlashGrpcServerHolder.cpp:206:17 (gtests_dbms+0xd3c45cf)
[2023-12-11T18:31:39.586Z] #5 std::__1::__unique_ifDB::FlashGrpcServerHolder::__unique_single std::__1::make_unique<DB::FlashGrpcServerHolder, DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig&, std::__1::shared_ptrDB::Logger const&>(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig&, std::__1::shared_ptrDB::Logger const&) /usr/local/bin/../include/c++/v1/__memory/unique_ptr.h:728:32 (gtests_dbms+0xd58ca69)
[2023-12-11T18:31:39.586Z] #6 DB::tests::MockComputeServerManager::startServers(std::__1::shared_ptrDB::Logger const&, int) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Debug/MockComputeServerManager.cpp:70:40 (gtests_dbms+0xd58ca69)
[2023-12-11T18:31:39.586Z] #7 DB::tests::MPPTaskTestUtils::startServers(unsigned long) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/MPPTaskTestUtils.cpp:85:42 (gtests_dbms+0x62134e4)
[2023-12-11T18:31:39.586Z] #8 DB::tests::ComputeServerRunner_multipleQuery_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/tests/gtest_compute_server.cpp:690:5 (gtests_dbms+0x312092a)
[2023-12-11T18:31:39.586Z] #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc366d75)
[2023-12-11T18:31:39.586Z] #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc366d75)
[2023-12-11T18:31:39.586Z] #11 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xc340915)
[2023-12-11T18:31:39.586Z] #12 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xc342144)
[2023-12-11T18:31:39.586Z] #13 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xc342e0d)
[2023-12-11T18:31:39.586Z] #14 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xc352fc0)
[2023-12-11T18:31:39.586Z] #15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc368215)
[2023-12-11T18:31:39.586Z] #16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc368215)
[2023-12-11T18:31:39.586Z] #17 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xc3522a8)
[2023-12-11T18:31:39.586Z] #18 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x5f70f5e)
[2023-12-11T18:31:39.586Z] #19 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x5f70f5e)
[2023-12-11T18:31:39.586Z]
[2023-12-11T18:31:39.586Z] Thread T1346 'async_poller' (tid=7796, running) created by main thread at:
[2023-12-11T18:31:39.586Z] #0 pthread_create /llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:977:3 (gtests_dbms+0x2b8717d)
[2023-12-11T18:31:39.586Z] #1 std::__1::__libcpp_thread_create(unsigned long*, void* ()(void), void*) /usr/local/bin/../include/c++/v1/__threading_support:514:10 (gtests_dbms+0xd3c45cf)
[2023-12-11T18:31:39.587Z] #2 std::__1::thread::thread<std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&)::'lambda'(auto&&...)&, void>(DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&) /usr/local/bin/../include/c++/v1/thread:307:16 (gtests_dbms+0xd3c45cf)
[2023-12-11T18:31:39.587Z] #3 std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_8&&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/ThreadFactory.h:50:16 (gtests_dbms+0xd3c45cf)
[2023-12-11T18:31:39.587Z] #4 DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Server/FlashGrpcServerHolder.cpp:206:17 (gtests_dbms+0xd3c45cf)
[2023-12-11T18:31:39.587Z] #5 std::__1::__unique_ifDB::FlashGrpcServerHolder::__unique_single std::__1::make_unique<DB::FlashGrpcServerHolder, DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig&, std::__1::shared_ptrDB::Logger const&>(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig&, std::__1::shared_ptrDB::Logger const&) /usr/local/bin/../include/c++/v1/__memory/unique_ptr.h:728:32 (gtests_dbms+0xd58ca69)
[2023-12-11T18:31:39.587Z] #6 DB::tests::MockComputeServerManager::startServers(std::__1::shared_ptrDB::Logger const&, int) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Debug/MockComputeServerManager.cpp:70:40 (gtests_dbms+0xd58ca69)
[2023-12-11T18:31:39.587Z] #7 DB::tests::MPPTaskTestUtils::startServers(unsigned long) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/MPPTaskTestUtils.cpp:85:42 (gtests_dbms+0x62134e4)
[2023-12-11T18:31:39.587Z] #8 DB::tests::ComputeServerRunner_multipleQuery_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/tests/gtest_compute_server.cpp:690:5 (gtests_dbms+0x312092a)
[2023-12-11T18:31:39.587Z] #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc366d75)
[2023-12-11T18:31:39.587Z] #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc366d75)
[2023-12-11T18:31:39.587Z] #11 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xc340915)
[2023-12-11T18:31:39.587Z] #12 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xc342144)
[2023-12-11T18:31:39.587Z] #13 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xc342e0d)
[2023-12-11T18:31:39.587Z] #14 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xc352fc0)
[2023-12-11T18:31:39.587Z] #15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc368215)
[2023-12-11T18:31:39.587Z] #16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc368215)
[2023-12-11T18:31:39.587Z] #17 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xc3522a8)
[2023-12-11T18:31:39.587Z] #18 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x5f70f5e)
[2023-12-11T18:31:39.587Z] #19 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x5f70f5e)
[2023-12-11T18:31:39.587Z]
[2023-12-11T18:31:39.587Z] SUMMARY: ThreadSanitizer: data race /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/protobuf/src/google/protobuf/generated_message_reflection.cc:3007:10 in google::protobuf::internal::AssignDescriptors(google::protobuf::internal::DescriptorTable const* ()(), std::__1::once_flag, google::protobuf::Metadata const&)

@yibin87 yibin87 changed the title TSan new failed compute test case ComputeServerRunner.aggWithColumnPrune TSan new failed compute test case Dec 12, 2023
@yibin87 yibin87 changed the title TSan new failed compute test case TSan failed compute test case Dec 12, 2023
@yibin87 yibin87 mentioned this issue Dec 12, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 13, 2023
@yibin87
Add protobuf::internal::AssignDescriptors into tsan.suppression to ig…
5417d84 
…nore false positive warning (#8499)

ref #8285
This was referenced Dec 15, 2023
Closed
Merged
ti-chi-bot bot pushed a commit that referenced this issue Dec 15, 2023
@yibin87
Update tsan suppressions to ignore protobuf DebugString and ShortDebu…
9e1cded 
…gString (#8530)

ref #8285
@yibin87 yibin87 mentioned this issue Dec 19, 2023
Merged
11 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 19, 2023
@yibin87
Add mutex to protect concurrent prepareMPPTasks (#8553)
c15fcc2 
ref #8285
@yibin87 yibin87 mentioned this issue Dec 22, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 22, 2023
@yibin87
Fix gtest data race detected by tsan check (#8565)
ddf3f97 
ref #8285
JaySon-Huang pushed a commit to JaySon-Huang/tiflash that referenced this issue Dec 27, 2023
@yibin87 @JaySon-Huang
Fix gtest data race detected by tsan check (pingcap#8565)
8f5b7f0 
ref pingcap#8285
@yibin87 yibin87 changed the title TSan failed compute test case ASAN/TSan failed compute test case Feb 22, 2024
@yibin87
Copy link
Contributor Author

yibin87 commented Feb 22, 2024

2024-02-21T19:09:49.037Z] =================================================================
[2024-02-21T19:09:49.037Z] ==68367==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffe4b219334 at pc 0x00000ff6e9db bp 0x7ffe4b219250 sp 0x7ffe4b219248
[2024-02-21T19:09:49.037Z] READ of size 4 at 0x7ffe4b219334 thread T0
[2024-02-21T19:09:49.037Z] #0 0xff6e9da in DB::DecimalField<DB::Decimal >::operator double<double, (void*)0>() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Core/Field.h:101:12
[2024-02-21T19:09:49.037Z] #1 0xff6e9da in double DB::TiDBConvertToFloat<DB::DataTypeDecimal<DB::Decimal >, DB::DataTypeNumber, true, false>::toFloat<DB::Decimal >(DB::DecimalField<DB::Decimal > const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Functions/FunctionsTiDBConversion.h:666:37
[2024-02-21T19:09:49.037Z] #2 0xff6e9da in DB::TiDBConvertToFloat<DB::DataTypeDecimal<DB::Decimal >, DB::DataTypeNumber, true, false>::execute(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator > const&, unsigned long, bool, tipb::FieldType const&, DB::Context const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Functions/FunctionsTiDBConversion.h:778:29
[2024-02-21T19:09:49.037Z] #3 0xfbc19a6 in std::__1::__function::__value_func<void (DB::Block&, std::__1::vector<unsigned long, std::__1::allocator > const&, unsigned long, bool, tipb::FieldType const&, DB::Context const&)>::operator()(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator > const&, unsigned long&&, bool&&, tipb::FieldType const&, DB::Context const&) const /usr/local/bin/../include/c++/v1/__functional/function.h:498:16
[2024-02-21T19:09:49.037Z] #4 0xfbc19a6 in std::__1::function<void (DB::Block&, std::__1::vector<unsigned long, std::__1::allocator > const&, unsigned long, bool, tipb::FieldType const&, DB::Context const&)>::operator()(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator > const&, unsigned long, bool, tipb::FieldType const&, DB::Context const&) const /usr/local/bin/../include/c++/v1/__functional/function.h:1175:12
[2024-02-21T19:09:49.037Z] #5 0xfbc19a6 in DB::ExecutableFunctionTiDBCast<>::executeImpl(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator > const&, unsigned long) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Functions/FunctionsTiDBConversion.h:1797:9
[2024-02-21T19:09:49.037Z] #6 0x2092ad5e in DB::IExecutableFunction::execute(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator > const&, unsigned long) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Functions/IFunction.cpp:235:5
[2024-02-21T19:09:49.037Z] #7 0x2092ad5e in DB::IExecutableFunction::defaultImplementationForConstantArguments(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator > const&, unsigned long) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Functions/IFunction.cpp:194:5
[2024-02-21T19:09:49.037Z] #8 0x2092ba5b in DB::IExecutableFunction::execute(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator > const&, unsigned long) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Functions/IFunction.cpp:229:9
[2024-02-21T19:09:49.037Z] #9 0x7648f0a in DB::IFunctionBase::execute(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator > const&, unsigned long) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Functions/IFunction.h:106:32
[2024-02-21T19:09:49.037Z] #10 0x1ded6c2d in DB::ExpressionAction::prepare(DB::Block&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Interpreters/ExpressionActions.cpp:194:23
[2024-02-21T19:09:49.037Z] #11 0x1dee0d0e in DB::ExpressionActions::addImpl(DB::ExpressionAction, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > >&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Interpreters/ExpressionActions.cpp:527:12
[2024-02-21T19:09:49.037Z] #12 0x1dee131d in DB::ExpressionActions::add(DB::ExpressionAction const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Interpreters/ExpressionActions.cpp:500:5
[2024-02-21T19:09:49.037Z] #13 0x20e78f52 in DB::DAGExpressionAnalyzerHelper::buildCastFunctionInternal(DB::DAGExpressionAnalyzer*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > > const&, bool, tipb::FieldType const&, std::__1::shared_ptrDB::ExpressionActions const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Coprocessor/DAGExpressionAnalyzerHelper.cpp:259:14
[2024-02-21T19:09:49.037Z] #14 0x20e799ca in DB::DAGExpressionAnalyzerHelper::buildCastFunction(DB::DAGExpressionAnalyzer*, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Coprocessor/DAGExpressionAnalyzerHelper.cpp:281:12
[2024-02-21T19:09:49.037Z] #15 0x20e8ec5a in decltype(static_cast<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > (&>(fp)(static_castDB::DAGExpressionAnalyzer*(fp0), static_cast<tipb::Expr const&>(fp0), static_cast<std::__1::shared_ptrDB::ExpressionActions const&>(fp0))) std::__1::__invoke<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > (&)(DB::DAGExpressionAnalyzer*, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&), DB::DAGExpressionAnalyzer*, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&>(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > (&)(DB::DAGExpressionAnalyzer, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&), DB::DAGExpressionAnalyzer*&&, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&) /usr/local/bin/../include/c++/v1/type_traits:3918:1
[2024-02-21T19:09:49.037Z] #16 0x20e8ec5a in std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > std::__1::__invoke_void_return_wrapper<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, false>::__call<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > (&)(DB::DAGExpressionAnalyzer, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&), DB::DAGExpressionAnalyzer*, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&>(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > (&)(DB::DAGExpressionAnalyzer, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&), DB::DAGExpressionAnalyzer*&&, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:30:16
[2024-02-21T19:09:49.037Z] #17 0x20e8ec5a in std::__1::__function::__alloc_func<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > ()(DB::DAGExpressionAnalyzer, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&), std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > ()(DB::DAGExpressionAnalyzer, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&)>, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > (DB::DAGExpressionAnalyzer*, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&)>::operator()(DB::DAGExpressionAnalyzer*&&, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&) /usr/local/bin/../include/c++/v1/__functional/function.h:171:16
[2024-02-21T19:09:49.038Z] #18 0x20e8ec5a in std::__1::__function::__func<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > ()(DB::DAGExpressionAnalyzer, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&), std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > ()(DB::DAGExpressionAnalyzer, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&)>, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > (DB::DAGExpressionAnalyzer*, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&)>::operator()(DB::DAGExpressionAnalyzer*&&, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&) /usr/local/bin/../include/c++/v1/__functional/function.h:345:12
[2024-02-21T19:09:49.038Z] #19 0x20e802d8 in std::__1::__function::__value_func<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > (DB::DAGExpressionAnalyzer*, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&)>::operator()(DB::DAGExpressionAnalyzer*&&, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&) const /usr/local/bin/../include/c++/v1/__functional/function.h:498:16
[2024-02-21T19:09:49.038Z] #20 0x20e802d8 in std::__1::function<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > (DB::DAGExpressionAnalyzer*, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&)>::operator()(DB::DAGExpressionAnalyzer*, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&) const /usr/local/bin/../include/c++/v1/__functional/function.h:1175:12
[2024-02-21T19:09:49.038Z] #21 0x20e802d8 in DB::DAGExpressionAnalyzerHelper::buildFunction(DB::DAGExpressionAnalyzer*, tipb::Expr const&, std::__1::shared_ptrDB::ExpressionActions const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Coprocessor/DAGExpressionAnalyzerHelper.cpp:534:16
[2024-02-21T19:09:49.038Z] #22 0xd820ea7 in DB::tests::buildFunction(DB::Context&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::vector<unsigned long, std::__1::allocator > const&, std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocatorDB::ColumnWithTypeAndName > const&, TiDB::ITiDBCollator const* const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/FunctionTestUtils.cpp:314:24
[2024-02-21T19:09:49.038Z] #23 0xd822f26 in DB::tests::executeFunction(DB::Context&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::vector<unsigned long, std::__1::allocator > const&, std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocatorDB::ColumnWithTypeAndName > const&, TiDB::ITiDBCollator const* const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/FunctionTestUtils.cpp:405:11
[2024-02-21T19:09:49.038Z] #24 0xd8228d2 in DB::tests::executeFunction(DB::Context&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocatorDB::ColumnWithTypeAndName > const&, TiDB::ITiDBCollator const* const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/FunctionTestUtils.cpp:363:12
[2024-02-21T19:09:49.038Z] #25 0xd832255 in DB::tests::FunctionTest::executeFunction(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocatorDB::ColumnWithTypeAndName > const&, TiDB::ITiDBCollator const* const&, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/FunctionTestUtils.cpp:571:12
[2024-02-21T19:09:49.038Z] #26 0xb0bddea in DB::tests::(anonymous namespace)::TestTidbConversion_castDecimalAsRealBasic_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Functions/tests/gtest_tidb_conversion.cpp:1595:5
[2024-02-21T19:09:49.038Z] #27 0x1bc1ce30 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10
[2024-02-21T19:09:49.038Z] #28 0x1bc1ce30 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14
[2024-02-21T19:09:49.038Z] #29 0x1bbbb5d8 in testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5
[2024-02-21T19:09:49.038Z] #30 0x1bbbe767 in testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11
[2024-02-21T19:09:49.038Z] #31 0x1bbbfcdf in testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28
[2024-02-21T19:09:49.038Z] #32 0x1bbea6f8 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43
[2024-02-21T19:09:49.038Z] #33 0x1bc1f430 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10
[2024-02-21T19:09:49.038Z] #34 0x1bc1f430 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14
[2024-02-21T19:09:49.038Z] #35 0x1bbe8f71 in testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10
[2024-02-21T19:09:49.038Z] #36 0xd166d4a in RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46
[2024-02-21T19:09:49.038Z] #37 0xd166d4a in main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16
[2024-02-21T19:09:49.038Z] #38 0x7f60a4637554 in __libc_start_main (/lib64/libc.so.6+0x22554)
[2024-02-21T19:09:49.038Z] #39 0x4cf8fe8 in _start (/tiflash/gtests_dbms+0x4cf8fe8)
[2024-02-21T19:09:49.038Z]
[2024-02-21T19:09:49.038Z] Address 0x7ffe4b219334 is located in stack of thread T0 at offset 212 in frame
[2024-02-21T19:09:49.038Z] #0 0xff6dd7f in DB::TiDBConvertToFloat<DB::DataTypeDecimal<DB::Decimal >, DB::DataTypeNumber, true, false>::execute(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator > const&, unsigned long, bool, tipb::FieldType const&, DB::Context const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Functions/FunctionsTiDBConversion.h:754
[2024-02-21T19:09:49.038Z]
[2024-02-21T19:09:49.038Z] This frame has 7 object(s):
[2024-02-21T19:09:49.038Z] [32, 56) 'ref.tmp.i.i'
[2024-02-21T19:09:49.038Z] [96, 97) 'x.addr.i.i79'
[2024-02-21T19:09:49.038Z] [112, 120) 'x.addr.i.i'
[2024-02-21T19:09:49.038Z] [144, 152) 'col_to' (line 758)
[2024-02-21T19:09:49.038Z] [176, 184) 'col_null_map_to' (line 761)
[2024-02-21T19:09:49.038Z] [208, 288) 'ref.tmp18' (line 777) <== Memory access at offset 212 is inside this variable
[2024-02-21T19:09:49.038Z] [320, 328) 'ref.tmp32' (line 856)
[2024-02-21T19:09:49.038Z] HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
[2024-02-21T19:09:49.038Z] (longjmp and C++ exceptions are supported)
[2024-02-21T19:09:49.038Z] SUMMARY: AddressSanitizer: stack-use-after-scope /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Core/Field.h:101:12 in DB::DecimalField<DB::Decimal >::operator double<double, (void*)0>() const
[2024-02-21T19:09:49.038Z] Shadow bytes around the buggy address:
[2024-02-21T19:09:49.038Z] 0x10004963b210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[2024-02-21T19:09:49.038Z] 0x10004963b220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[2024-02-21T19:09:49.038Z] 0x10004963b230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[2024-02-21T19:09:49.038Z] 0x10004963b240: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
[2024-02-21T19:09:49.038Z] 0x10004963b250: f8 f8 f8 f2 f2 f2 f2 f2 f8 f2 f8 f2 f2 f2 00 f2
[2024-02-21T19:09:49.038Z] =>0x10004963b260: f2 f2 00 f2 f2 f2[f8]f8 f8 f8 f8 f8 f8 f8 f8 f8
[2024-02-21T19:09:49.038Z] 0x10004963b270: f2 f2 f2 f2 f8 f3 f3 f3 00 00 00 00 00 00 00 00
[2024-02-21T19:09:49.038Z] 0x10004963b280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[2024-02-21T19:09:49.038Z] 0x10004963b290: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
[2024-02-21T19:09:49.038Z] 0x10004963b2a0: 00 f2 f2 f2 01 f2 00 00 00 f3 f3 f3 f3 f3 f3 f3
[2024-02-21T19:09:49.038Z] 0x10004963b2b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[2024-02-21T19:09:49.038Z] Shadow byte legend (one shadow byte represents 8 application bytes):
[2024-02-21T19:09:49.038Z] Addressable: 00
[2024-02-21T19:09:49.038Z] Partially addressable: 01 02 03 04 05 06 07
[2024-02-21T19:09:49.038Z] Heap left redzone: fa
[2024-02-21T19:09:49.038Z] Freed heap region: fd
[2024-02-21T19:09:49.038Z] Stack left redzone: f1
[2024-02-21T19:09:49.038Z] Stack mid redzone: f2
[2024-02-21T19:09:49.038Z] Stack right redzone: f3
[2024-02-21T19:09:49.038Z] Stack after return: f5
[2024-02-21T19:09:49.038Z] Stack use after scope: f8
[2024-02-21T19:09:49.038Z] Global redzone: f9
[2024-02-21T19:09:49.038Z] Global init order: f6
[2024-02-21T19:09:49.038Z] Poisoned by user: f7
[2024-02-21T19:09:49.038Z] Container overflow: fc
[2024-02-21T19:09:49.038Z] Array cookie: ac
[2024-02-21T19:09:49.038Z] Intra object redzone: bb
[2024-02-21T19:09:49.038Z] ASan internal: fe
[2024-02-21T19:09:49.038Z] Left alloca redzone: ca
[2024-02-21T19:09:49.038Z] Right alloca redzone: cb
[2024-02-21T19:09:49.038Z] ==68367==ABORTING

@yibin87 yibin87 mentioned this issue Feb 22, 2024
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Feb 22, 2024
@yibin87
Fix decimal to real asan stack-use-after-scope error (#8780)
cab8352 
ref #8285
@yibin87
Copy link
Contributor Author

yibin87 commented Jul 8, 2024

New testCancelMPPGather case failure due to sleep too short:
[2024-07-07T18:31:42.163Z] [2024/07/08 02:31:36.084 +08:00] [ERROR] [Exception.cpp:96] ["std::exception. Code: 1001, type: testing::AssertionException, e.what() = /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/tests/gtest_compute_server.cpp:1003: Failure\nValue of: TiFlashMetrics::instance() .tiflash_task_scheduler.get(tiflash_task_scheduler_metrics::type_active_queries_count, "") .Value() == 2\n Actual: false\nExpected: true"] [source="virtual void DB::tests::ComputeServerRunner_testMinTSOActiveSetSoftLimit_Test::TestBody()"] [thread_id=1]

@yibin87 yibin87 mentioned this issue Jul 8, 2024
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jul 8, 2024
@yibin87
Change sleep time for sanitizer build type of gtest_compute_server (#…
ce637fe 
…9194)

ref #8285

Signed-off-by: yibin <[email protected]>
@yibin87 yibin87 mentioned this issue Jul 9, 2024
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jul 9, 2024
Adjust sanitizer sleep time for cancelMPPGather (#9200)
cd2d7e6 
ref #8285

In #9194, sleep time is changed to 8s in sanitizer mode. However, the case still failed occasionally. So change to a larger time span.

Signed-off-by: yibin <[email protected]>
@yibin87
Copy link
Contributor Author

yibin87 commented Jul 11, 2024

Group/StoreIngestTest.ConcurrentIngestAndWrite/1
ThreadSanitizer: CHECK failed: sanitizer_deadlock_detector.h:67 "((n_all_locks_)) < (((sizeof(all_locks_with_contexts_)/sizeof((all_locks_with_contexts_)[0]))))" (0x40, 0x40) (tid=1998041)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/compute priority/P2 The issue has P2 priority. severity/minor type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@yibin87