Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tsan: data race on ComputeServerRunner.cancelJoinTasks #7432

Closed
SeaRise opened this issue May 7, 2023 · 1 comment · Fixed by #7434
Closed

Tsan: data race on ComputeServerRunner.cancelJoinTasks #7432

SeaRise opened this issue May 7, 2023 · 1 comment · Fixed by #7434
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. component/compute report/customer Customers have encountered this bug. severity/major type/bug The issue is confirmed as a bug.

Comments

@SeaRise
Copy link
Contributor

SeaRise commented May 7, 2023 

[2023-05-06T18:09:32.507Z] ==================
[2023-05-06T18:09:32.507Z] WARNING: ThreadSanitizer: data race (pid=37941)
[2023-05-06T18:09:32.507Z]   Read of size 8 at 0x7b0c00167ac0 by thread T176:
[2023-05-06T18:09:32.507Z]     #0 memcpy /llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:827:5 (gtests_dbms+0x27bb5e0)
[2023-05-06T18:09:32.507Z]     #1 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__assign_external(char const*, unsigned long) string.cpp (libc++.so.1+0x69254)
[2023-05-06T18:09:32.507Z]     #2 DB::MPPTask::runImpl() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Mpp/MPPTask.cpp:484:25 (gtests_dbms+0xdb9d53f)
[2023-05-06T18:09:32.507Z]     #3 DB::MPPTask::run()::$_17::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Mpp/MPPTask.cpp:154:97 (gtests_dbms+0xdb9f0a3)
[2023-05-06T18:09:32.507Z]     #4 decltype(static_cast<DB::MPPTask::run()::$_17&>(fp)()) std::__1::__invoke<DB::MPPTask::run()::$_17&>(DB::MPPTask::run()::$_17&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xdb9f0a3)
[2023-05-06T18:09:32.507Z]     #5 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::MPPTask::run()::$_17&>(DB::MPPTask::run()::$_17&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0xdb9f0a3)
[2023-05-06T18:09:32.507Z]     #6 std::__1::__function::__alloc_func<DB::MPPTask::run()::$_17, std::__1::allocator<DB::MPPTask::run()::$_17>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0xdb9f0a3)
[2023-05-06T18:09:32.507Z]     #7 std::__1::__function::__func<DB::MPPTask::run()::$_17, std::__1::allocator<DB::MPPTask::run()::$_17>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0xdb9f0a3)
[2023-05-06T18:09:32.507Z]     #8 std::__1::__function::__value_func<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xe319f1d)
[2023-05-06T18:09:32.507Z]     #9 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xe319f1d)
[2023-05-06T18:09:32.507Z]     #10 decltype(static_cast<std::__1::function<void ()> const&>(fp)()) std::__1::__invoke<std::__1::function<void ()> const&>(std::__1::function<void ()> const&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xe319f1d)
[2023-05-06T18:09:32.507Z]     #11 std::__1::invoke_result<std::__1::function<void ()> const&>::type std::__1::invoke<std::__1::function<void ()> const&>(std::__1::function<void ()> const&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:93:12 (gtests_dbms+0xe319f1d)
[2023-05-06T18:09:32.507Z]     #12 auto std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>&&)::'lambda'(auto&&...)::operator()<>(auto&&...) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/ThreadFactory.h:46:20 (gtests_dbms+0xe319f1d)
[2023-05-06T18:09:32.507Z]     #13 decltype(static_cast<std::__1::function<void ()>>(fp)()) std::__1::__invoke<std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>&&)::'lambda'(auto&&...)>(std::__1::function<void ()>&&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xe319ba7)
[2023-05-06T18:09:32.507Z]     #14 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>&&)::'lambda'(auto&&...)>(std::__1::tuple<std::__1::function<void ()>, std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>&&)::'lambda'(auto&&...)>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xe319ba7)
[2023-05-06T18:09:32.507Z]     #15 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>&&)::'lambda'(auto&&...)> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xe319ba7)
[2023-05-06T18:09:32.507Z] 
[2023-05-06T18:09:32.507Z]   Previous write of size 8 at 0x7b0c00167ac0 by main thread:
[2023-05-06T18:09:32.507Z]     #0 operator new(unsigned long) /llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:64:3 (gtests_dbms+0x2826598)
[2023-05-06T18:09:32.507Z]     #1 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::operator=(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <null> (libc++.so.1+0x6a328)
[2023-05-06T18:09:32.507Z]     #2 DB::MPPTaskManager::abortMPPQuery(DB::MPPQueryId const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::AbortType) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Mpp/MPPTaskManager.cpp:201:20 (gtests_dbms+0xdba8d77)
[2023-05-06T18:09:32.507Z]     #3 DB::FlashService::cancelMPPTaskForTest(mpp::CancelTaskRequest const*, mpp::CancelTaskResponse*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/FlashService.cpp:518:19 (gtests_dbms+0xd9c9a7d)
[2023-05-06T18:09:32.507Z]     #4 DB::tests::MockComputeServerManager::cancelQuery(DB::MPPQueryId const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Debug/MockComputeServerManager.cpp:133:40 (gtests_dbms+0xcad03be)
[2023-05-06T18:09:32.507Z]     #5 DB::tests::ComputeServerRunner_cancelJoinTasks_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/tests/gtest_compute_server.cpp:545:46 (gtests_dbms+0x2c9693c)
[2023-05-06T18:09:32.507Z]     #6 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xb63e885)
[2023-05-06T18:09:32.508Z]     #7 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xb63e885)
[2023-05-06T18:09:32.508Z]     #8 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xb618425)
[2023-05-06T18:09:32.508Z]     #9 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xb619c54)
[2023-05-06T18:09:32.508Z]     #10 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xb61a91d)
[2023-05-06T18:09:32.508Z]     #11 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xb62aad0)
[2023-05-06T18:09:32.508Z]     #12 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xb63fd25)
[2023-05-06T18:09:32.508Z]     #13 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xb63fd25)
[2023-05-06T18:09:32.508Z]     #14 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xb629db8)
[2023-05-06T18:09:32.508Z]     #15 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x59356e6)
[2023-05-06T18:09:32.508Z]     #16 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:115:16 (gtests_dbms+0x59356e6)
[2023-05-06T18:09:32.508Z] 
[2023-05-06T18:09:32.508Z]   Location is heap block of size 48 at 0x7b0c00167ac0 allocated by main thread:
[2023-05-06T18:09:32.508Z]     #0 operator new(unsigned long) /llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:64:3 (gtests_dbms+0x2826598)
[2023-05-06T18:09:32.508Z]     #1 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::operator=(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <null> (libc++.so.1+0x6a328)
[2023-05-06T18:09:32.508Z]     #2 DB::MPPTaskManager::abortMPPQuery(DB::MPPQueryId const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::AbortType) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Mpp/MPPTaskManager.cpp:201:20 (gtests_dbms+0xdba8d77)
[2023-05-06T18:09:32.508Z]     #3 DB::FlashService::cancelMPPTaskForTest(mpp::CancelTaskRequest const*, mpp::CancelTaskResponse*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/FlashService.cpp:518:19 (gtests_dbms+0xd9c9a7d)
[2023-05-06T18:09:32.508Z]     #4 DB::tests::MockComputeServerManager::cancelQuery(DB::MPPQueryId const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Debug/MockComputeServerManager.cpp:133:40 (gtests_dbms+0xcad03be)
[2023-05-06T18:09:32.508Z]     #5 DB::tests::ComputeServerRunner_cancelJoinTasks_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/tests/gtest_compute_server.cpp:545:46 (gtests_dbms+0x2c9693c)
@SeaRise SeaRise added type/bug The issue is confirmed as a bug. component/compute severity/moderate labels May 7, 2023
@SeaRise SeaRise mentioned this issue May 8, 2023
Merged
12 tasks
@ti-chi-bot ti-chi-bot bot closed this as completed in #7434 May 8, 2023
ti-chi-bot bot pushed a commit that referenced this issue May 8, 2023
@SeaRise
Tsan: fix data race on ComputeServerRunner.cancelJoinTasks (#7434)
12bda10 
close #7432
@JaySon-Huang JaySon-Huang added affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. labels Jan 24, 2024
This was referenced Jan 24, 2024
Merged
Closed
@ti-chi-bot ti-chi-bot mentioned this issue Jan 24, 2024
Merged
12 tasks
@SeaRise SeaRise removed the affects-6.1 This bug affects the 6.1.x(LTS) versions. label Jan 24, 2024
ti-chi-bot bot pushed a commit that referenced this issue Jan 24, 2024
@ti-chi-bot
Tsan: fix data race on ComputeServerRunner.cancelJoinTasks (#7434) (#…
3131f5a 
…8727)

close #7432
ti-chi-bot bot pushed a commit that referenced this issue Jan 24, 2024
@ti-chi-bot
Tsan: fix data race on ComputeServerRunner.cancelJoinTasks (#7434) (#…
9e5b582 
…8729)

close #7432
@seiya-annie
Copy link

/found customer

@ti-chi-bot ti-chi-bot bot added the report/customer Customers have encountered this bug. label Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. component/compute report/customer Customers have encountered this bug. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Tsan: fix data race on ComputeServerRunner.cancelJoinTasks SeaRise/tiflash
3 participants
@JaySon-Huang @SeaRise @seiya-annie