Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ThreadSanitizer detect potential deadlock in AsyncTasksTest #8624

Closed
JaySon-Huang opened this issue Dec 29, 2023 · 1 comment · Fixed by #8614
Closed

ThreadSanitizer detect potential deadlock in AsyncTasksTest #8624

JaySon-Huang opened this issue Dec 29, 2023 · 1 comment · Fixed by #8614
Assignees
@CalvinNeo
Labels
component/storage severity/major type/bug The issue is confirmed as a bug.

Comments

@JaySon-Huang
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

https://ci.pingcap.net/blue/organizations/jenkins/tiflash-sanitizer-daily/detail/tiflash-sanitizer-daily/1613/pipeline/

/tiflash/gtests_dbms RegionKVStoreOldTest.ReadIndex
/tiflash/gtests_dbms AsyncTasksTest.AsyncTasksNormal
/tiflash/gtests_dbms RegionKVStoreV2Test.KVStoreSingleSnap4

[2023-12-28T18:33:49.691Z] �[0;33mNote: Google Test filter = AsyncTasksTest.AsyncTasksNormal
[2023-12-28T18:33:49.691Z] �[m�[0;32m[==========] �[mRunning 1 test from 1 test case.
[2023-12-28T18:33:49.691Z] �[0;32m[----------] �[mGlobal test environment set-up.
[2023-12-28T18:33:49.691Z] �[0;32m[----------] �[m1 test from AsyncTasksTest
[2023-12-28T18:33:49.691Z] �[0;32m[ RUN      ] �[mAsyncTasksTest.AsyncTasksNormal
[2023-12-28T18:33:49.691Z] [2023/12/29 02:33:45.464 +08:00] [INFO] [gtests_async_tasks.cpp:29] ["Cancel and addTask"] [thread_id=1]
[2023-12-28T18:33:49.691Z] [2023/12/29 02:33:45.466 +08:00] [INFO] [AsyncTasks.h:56] ["Pending 0 tasks when destructing"] [thread_id=1]
[2023-12-28T18:33:49.691Z] [2023/12/29 02:33:45.466 +08:00] [INFO] [gtests_async_tasks.cpp:56] ["Lifetime of tasks"] [thread_id=1]
[2023-12-28T18:33:49.691Z] [2023/12/29 02:33:45.466 +08:00] [DEBUG] [Ctl.cpp:34] ["Enabled: after_AsyncTasks::addTask_scheduled"] [source=SyncPointCtl] [thread_id=1]
[2023-12-28T18:33:49.691Z] [2023/12/29 02:33:45.466 +08:00] [DEBUG] [Ctl.cpp:34] ["Enabled: before_AsyncTasks::addTask_quit"] [source=SyncPointCtl] [thread_id=1]
[2023-12-28T18:33:49.692Z] [2023/12/29 02:33:45.467 +08:00] [DEBUG] [Ctl.cpp:69] ["waitAndPause(after_AsyncTasks::addTask_scheduled) waiting..."] [source=SyncPointCtl] [thread_id=98]
[2023-12-28T18:33:49.692Z] [2023/12/29 02:33:45.468 +08:00] [DEBUG] [Ctl.cpp:69] ["waitAndPause(before_AsyncTasks::addTask_quit) waiting..."] [source=SyncPointCtl] [thread_id=99]
[2023-12-28T18:33:49.692Z] [2023/12/29 02:33:46.208 +08:00] [DEBUG] [Ctl.cpp:86] ["SYNC_FOR(after_AsyncTasks::addTask_scheduled) trying... \n\n# Current Stack: \n       0x2bbc38b\t__interceptor_backtrace [gtests_dbms+45859723]\n                \t/llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:4202\n       0xfaba740\tStackTrace::StackTrace() [gtests_dbms+262907712]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0xfb4fef1\tDB::SyncPointCtl::sync(char const*) [gtests_dbms+263519985]\n                \tdbms/src/Common/SyncPoint/Ctl.cpp:86\n       0x5b00925\tDB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>) [gtests_dbms+95422757]\n                \tdbms/src/Storages/KVStore/Utils/AsyncTasks.h:274\n       0x5affa19\tDB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTask(unsigned long, std::__1::function<void ()>) [gtests_dbms+95418905]\n                \tdbms/src/Storages/KVStore/Utils/AsyncTasks.h:285\n       0x5af6543\tDB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody() [gtests_dbms+95380803]\n                \tdbms/src/Storages/KVStore/tests/gtests_async_tasks.cpp:75\n       0xc41fbd6\tvoid testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) [gtests_dbms+205650902]\n                \tcontrib/googletest/googletest/src/gtest.cc:2479\n       0xc3f9776\ttesting::Test::Run() [gtests_dbms+205494134]\n                \tcontrib/googletest/googletest/src/gtest.cc:2517\n       0xc3fafa5\ttesting::TestInfo::Run() [gtests_dbms+205500325]\n                \tcontrib/googletest/googletest/src/gtest.cc:2693\n       0xc3fbc6e\ttesting::TestCase::Run() [gtests_dbms+205503598]\n                \tcontrib/googletest/googletest/src/gtest.cc:2811\n       0xc40be21\ttesting::internal::UnitTestImpl::RunAllTests() [gtests_dbms+205569569]\n                \tcontrib/googletest/googletest/src/gtest.cc:5177\n       0xc421076\tbool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) [gtests_dbms+205656182]\n                \tcontrib/googletest/googletest/src/gtest.cc:2479\n       0xc40b109\ttesting::UnitTest::Run() [gtests_dbms+205566217]\n                \tcontrib/googletest/googletest/src/gtest.cc:4786\n       0x5fd310f\tmain [gtests_dbms+100479247]\n                \tdbms/src/TestUtils/gtests_dbms_main.cpp:116\n  0x7fbf10532555\t__libc_start_main [libc.so.6+140629]\n       0x2b751a9\t<unknown symbol> [gtests_dbms+45568425]"] [source=SyncPointCtl] [thread_id=1]
[2023-12-28T18:33:49.692Z] [2023/12/29 02:33:46.208 +08:00] [DEBUG] [Ctl.cpp:92] ["SYNC_FOR(after_AsyncTasks::addTask_scheduled) matched waitAndPause(), paused until calling next()..."] [source=SyncPointCtl] [thread_id=1]
[2023-12-28T18:33:49.692Z] [2023/12/29 02:33:46.208 +08:00] [DEBUG] [Ctl.cpp:71] ["waitAndPause(after_AsyncTasks::addTask_scheduled) finished"] [source=SyncPointCtl] [thread_id=98]
[2023-12-28T18:33:49.693Z] [2023/12/29 02:33:46.208 +08:00] [DEBUG] [Ctl.cpp:77] ["next(after_AsyncTasks::addTask_scheduled) trying..."] [source=SyncPointCtl] [thread_id=98]
[2023-12-28T18:33:49.693Z] [2023/12/29 02:33:46.208 +08:00] [DEBUG] [Ctl.cpp:96] ["SYNC_FOR(after_AsyncTasks::addTask_scheduled) done"] [source=SyncPointCtl] [thread_id=1]
[2023-12-28T18:33:49.693Z] [2023/12/29 02:33:46.208 +08:00] [DEBUG] [Ctl.cpp:79] ["next(after_AsyncTasks::addTask_scheduled) done"] [source=SyncPointCtl] [thread_id=98]
[2023-12-28T18:33:49.693Z] [2023/12/29 02:33:46.208 +08:00] [DEBUG] [Ctl.cpp:50] ["Disabled: after_AsyncTasks::addTask_scheduled"] [source=SyncPointCtl] [thread_id=98]
[2023-12-28T18:33:49.693Z] [2023/12/29 02:33:46.209 +08:00] [DEBUG] [Ctl.cpp:86] ["SYNC_FOR(before_AsyncTasks::addTask_quit) trying... \n\n# Current Stack: \n       0x2bbc38b\t__interceptor_backtrace [gtests_dbms+45859723]\n                \t/llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:4202\n       0xfaba740\tStackTrace::StackTrace() [gtests_dbms+262907712]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0xfb4fef1\tDB::SyncPointCtl::sync(char const*) [gtests_dbms+263519985]\n                \tdbms/src/Common/SyncPoint/Ctl.cpp:86\n       0x5b00a57\tDB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>) [gtests_dbms+95423063]\n                \tdbms/src/Storages/KVStore/Utils/AsyncTasks.h:279\n       0x5affa19\tDB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTask(unsigned long, std::__1::function<void ()>) [gtests_dbms+95418905]\n                \tdbms/src/Storages/KVStore/Utils/AsyncTasks.h:285\n       0x5af6543\tDB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody() [gtests_dbms+95380803]\n                \tdbms/src/Storages/KVStore/tests/gtests_async_tasks.cpp:75\n       0xc41fbd6\tvoid testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) [gtests_dbms+205650902]\n                \tcontrib/googletest/googletest/src/gtest.cc:2479\n       0xc3f9776\ttesting::Test::Run() [gtests_dbms+205494134]\n                \tcontrib/googletest/googletest/src/gtest.cc:2517\n       0xc3fafa5\ttesting::TestInfo::Run() [gtests_dbms+205500325]\n                \tcontrib/googletest/googletest/src/gtest.cc:2693\n       0xc3fbc6e\ttesting::TestCase::Run() [gtests_dbms+205503598]\n                \tcontrib/googletest/googletest/src/gtest.cc:2811\n       0xc40be21\ttesting::internal::UnitTestImpl::RunAllTests() [gtests_dbms+205569569]\n                \tcontrib/googletest/googletest/src/gtest.cc:5177\n       0xc421076\tbool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) [gtests_dbms+205656182]\n                \tcontrib/googletest/googletest/src/gtest.cc:2479\n       0xc40b109\ttesting::UnitTest::Run() [gtests_dbms+205566217]\n                \tcontrib/googletest/googletest/src/gtest.cc:4786\n       0x5fd310f\tmain [gtests_dbms+100479247]\n                \tdbms/src/TestUtils/gtests_dbms_main.cpp:116\n  0x7fbf10532555\t__libc_start_main [libc.so.6+140629]\n       0x2b751a9\t<unknown symbol> [gtests_dbms+45568425]"] [source=SyncPointCtl] [thread_id=1]
[2023-12-28T18:33:49.694Z] [2023/12/29 02:33:46.209 +08:00] [DEBUG] [Ctl.cpp:92] ["SYNC_FOR(before_AsyncTasks::addTask_quit) matched waitAndPause(), paused until calling next()..."] [source=SyncPointCtl] [thread_id=1]
[2023-12-28T18:33:49.694Z] [2023/12/29 02:33:46.209 +08:00] [DEBUG] [Ctl.cpp:71] ["waitAndPause(before_AsyncTasks::addTask_quit) finished"] [source=SyncPointCtl] [thread_id=99]
[2023-12-28T18:33:49.694Z] [2023/12/29 02:33:46.209 +08:00] [DEBUG] [Ctl.cpp:77] ["next(before_AsyncTasks::addTask_quit) trying..."] [source=SyncPointCtl] [thread_id=99]
[2023-12-28T18:33:49.694Z] [2023/12/29 02:33:46.209 +08:00] [DEBUG] [Ctl.cpp:79] ["next(before_AsyncTasks::addTask_quit) done"] [source=SyncPointCtl] [thread_id=99]
[2023-12-28T18:33:49.694Z] [2023/12/29 02:33:46.209 +08:00] [DEBUG] [Ctl.cpp:96] ["SYNC_FOR(before_AsyncTasks::addTask_quit) done"] [source=SyncPointCtl] [thread_id=1]
[2023-12-28T18:33:49.694Z] [2023/12/29 02:33:46.209 +08:00] [DEBUG] [Ctl.cpp:50] ["Disabled: before_AsyncTasks::addTask_quit"] [source=SyncPointCtl] [thread_id=99]
[2023-12-28T18:33:49.694Z] [2023/12/29 02:33:46.260 +08:00] [INFO] [AsyncTasks.h:56] ["Pending 1 tasks when destructing"] [thread_id=1]
[2023-12-28T18:33:49.694Z] [2023/12/29 02:33:46.260 +08:00] [INFO] [gtests_async_tasks.cpp:82] ["Cancel in queue"] [thread_id=1]
[2023-12-28T18:33:49.694Z] ==================
[2023-12-28T18:33:49.694Z] WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=467053)
[2023-12-28T18:33:49.694Z]   Cycle in lock order graph: M52569 (0x7ffdc4b02ff8) => M52572 (0x7b10000b7718) => M52569
[2023-12-28T18:33:49.694Z] 
[2023-12-28T18:33:49.694Z]   Mutex M52572 acquired here while holding mutex M52569 in main thread:
[2023-12-28T18:33:49.694Z]     #0 pthread_mutex_lock /llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:4249:3 (gtests_dbms+0x2bbc7d8)
[2023-12-28T18:33:49.694Z]     #1 std::__1::mutex::lock() <null> (libc++.so.1+0x66f45)
[2023-12-28T18:33:49.694Z]     #2 DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTask(unsigned long, std::__1::function<void ()>) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/Utils/AsyncTasks.h:285:16 (gtests_dbms+0x5affa18)
[2023-12-28T18:33:49.694Z]     #3 DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/tests/gtests_async_tasks.cpp:90:34 (gtests_dbms+0x5af6966)
[2023-12-28T18:33:49.694Z]     #4 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc41fbd5)
[2023-12-28T18:33:49.694Z]     #5 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc41fbd5)
[2023-12-28T18:33:49.694Z]     #6 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xc3f9775)
[2023-12-28T18:33:49.694Z]     #7 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xc3fafa4)
[2023-12-28T18:33:49.694Z]     #8 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xc3fbc6d)
[2023-12-28T18:33:49.694Z]     #9 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xc40be20)
[2023-12-28T18:33:49.694Z]     #10 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc421075)
[2023-12-28T18:33:49.694Z]     #11 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc421075)
[2023-12-28T18:33:49.694Z]     #12 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xc40b108)
[2023-12-28T18:33:49.695Z]     #13 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x5fd310e)
[2023-12-28T18:33:49.695Z]     #14 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x5fd310e)
[2023-12-28T18:33:49.695Z] 
[2023-12-28T18:33:49.695Z]     Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
[2023-12-28T18:33:49.695Z] 
[2023-12-28T18:33:49.695Z]   Mutex M52569 acquired here while holding mutex M52572 in thread T105:
[2023-12-28T18:33:49.695Z]     #0 pthread_mutex_lock /llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:4249:3 (gtests_dbms+0x2bbc7d8)
[2023-12-28T18:33:49.695Z]     #1 std::__1::mutex::lock() <null> (libc++.so.1+0x66f45)
[2023-12-28T18:33:49.695Z]     #2 std::__1::__function::__value_func<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0x5b042c4)
[2023-12-28T18:33:49.695Z]     #3 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0x5b042c4)
[2023-12-28T18:33:49.695Z]     #4 decltype(static_cast<std::__1::function<void ()>&>(fp)()) std::__1::__invoke<std::__1::function<void ()>&>(std::__1::function<void ()>&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0x5b042c4)
[2023-12-28T18:33:49.695Z]     #5 std::__1::__packaged_task_func<std::__1::function<void ()>, std::__1::allocator<std::__1::function<void ()> >, void ()>::operator()() /usr/local/bin/../include/c++/v1/future:1687:12 (gtests_dbms+0x5b042c4)
[2023-12-28T18:33:49.695Z]     #6 std::__1::__packaged_task_function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/future:1869:12 (gtests_dbms+0x2caa2bd)
[2023-12-28T18:33:49.695Z]     #7 std::__1::packaged_task<void ()>::operator()() /usr/local/bin/../include/c++/v1/future:2089:9 (gtests_dbms+0x2caa2bd)
[2023-12-28T18:33:49.695Z]     #8 DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/Utils/AsyncTasks.h:270:13 (gtests_dbms+0x5b04738)
[2023-12-28T18:33:49.695Z]     #9 decltype(static_cast<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&>(fp)()) std::__1::__invoke<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&>(DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0x5b045dd)
[2023-12-28T18:33:49.695Z]     #10 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&>(DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0x5b045dd)
[2023-12-28T18:33:49.695Z]     #11 std::__1::__function::__alloc_func<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'(), std::__1::allocator<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0x5b045dd)
[2023-12-28T18:33:49.695Z]     #12 std::__1::__function::__func<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'(), std::__1::allocator<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0x5b045dd)
[2023-12-28T18:33:49.695Z]     #13 std::__1::__function::__value_func<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xfb14111)
[2023-12-28T18:33:49.695Z]     #14 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xfb14111)
[2023-12-28T18:33:49.695Z]     #15 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::worker(std::__1::__list_iterator<DB::ThreadFromGlobalPoolImpl<false>, void*>) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0xfb14111)
[2023-12-28T18:33:49.695Z]     #16 bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.695Z]     #17 decltype(static_cast<bool>(fp)()) std::__1::__invoke_constexpr<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&>(bool&&) /usr/local/bin/../include/c++/v1/type_traits:3924:1 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.696Z]     #18 decltype(auto) std::__1::__apply_tuple_impl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(bool&&, std::__1::tuple<>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/tuple:1536:1 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.696Z]     #19 decltype(auto) std::__1::apply<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(bool&&, std::__1::tuple<>&) /usr/local/bin/../include/c++/v1/tuple:1545:1 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.696Z]     #20 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()::operator()() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.h:236:17 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.696Z]     #21 decltype(static_cast<bool>(fp)()) std::__1::__invoke<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()&>(bool&&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.696Z]     #22 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()&>(DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.696Z]     #23 std::__1::__function::__alloc_func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.696Z]     #24 std::__1::__function::__func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.696Z]     #25 std::__1::__function::__value_func<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xfb1199a)
[2023-12-28T18:33:49.696Z]     #26 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xfb1199a)
[2023-12-28T18:33:49.696Z]     #27 DB::ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0xfb1199a)
[2023-12-28T18:33:49.696Z]     #28 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0xfb153b0)
[2023-12-28T18:33:49.696Z]     #29 decltype(static_cast<void>(fp)()) std::__1::__invoke<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xfb153b0)
[2023-12-28T18:33:49.696Z]     #30 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xfb153b0)
[2023-12-28T18:33:49.697Z]     #31 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xfb153b0)
[2023-12-28T18:33:49.697Z] 
[2023-12-28T18:33:49.697Z]   Thread T105 'ThreadPool' (tid=467165, running) created by main thread at:
[2023-12-28T18:33:49.697Z]     #0 pthread_create /llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:977:3 (gtests_dbms+0x2b9f4fd)
[2023-12-28T18:33:49.697Z]     #1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /usr/local/bin/../include/c++/v1/__threading_support:514:10 (gtests_dbms+0xfb14de8)
[2023-12-28T18:33:49.697Z]     #2 std::__1::thread::thread<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'(), void>(void&&) /usr/local/bin/../include/c++/v1/thread:307:16 (gtests_dbms+0xfb14de8)
[2023-12-28T18:33:49.697Z]     #3 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0xfb10008)
[2023-12-28T18:33:49.697Z]     #4 DB::ThreadPoolImpl<std::__1::thread>::scheduleOrThrow(std::__1::function<void ()>, long, unsigned long, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:198:5 (gtests_dbms+0xfb17ef4)
[2023-12-28T18:33:49.697Z]     #5 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.h:219:38 (gtests_dbms+0xfb17ef4)
[2023-12-28T18:33:49.697Z]     #6 bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0xfb12b80)
[2023-12-28T18:33:49.697Z]     #7 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::trySchedule(std::__1::function<void ()>, long, unsigned long) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:188:12 (gtests_dbms+0xfb1279a)
[2023-12-28T18:33:49.697Z]     #8 DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/Utils/AsyncTasks.h:273:33 (gtests_dbms+0x5b008c8)
[2023-12-28T18:33:49.697Z]     #9 DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTask(unsigned long, std::__1::function<void ()>) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/Utils/AsyncTasks.h:285:16 (gtests_dbms+0x5affa18)
[2023-12-28T18:33:49.697Z]     #10 DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/tests/gtests_async_tasks.cpp:37:22 (gtests_dbms+0x5af5cfa)
[2023-12-28T18:33:49.697Z]     #11 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc41fbd5)
[2023-12-28T18:33:49.697Z]     #12 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc41fbd5)
[2023-12-28T18:33:49.697Z]     #13 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xc3f9775)
[2023-12-28T18:33:49.697Z]     #14 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xc3fafa4)
[2023-12-28T18:33:49.697Z]     #15 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xc3fbc6d)
[2023-12-28T18:33:49.697Z]     #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xc40be20)
[2023-12-28T18:33:49.697Z]     #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc421075)
[2023-12-28T18:33:49.697Z]     #18 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc421075)
[2023-12-28T18:33:49.698Z]     #19 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xc40b108)
[2023-12-28T18:33:49.698Z]     #20 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x5fd310e)
[2023-12-28T18:33:49.698Z]     #21 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x5fd310e)
[2023-12-28T18:33:49.698Z] 
[2023-12-28T18:33:49.698Z] SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/usr/local/lib/x86_64-unknown-linux-gnu/libc++.so.1+0x66f45) in std::__1::mutex::lock()
[2023-12-28T18:33:49.698Z] ==================
[2023-12-28T18:33:49.698Z] LLVMSymbolizer: error reading file: No such file or directory
[2023-12-28T18:33:49.698Z] ==================
[2023-12-28T18:33:49.698Z] WARNING: ThreadSanitizer: data race (pid=467053)
[2023-12-28T18:33:49.698Z]   Write of size 1 at 0x7ffdc4b030a0 by thread T105:
[2023-12-28T18:33:49.698Z]     #0 DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody()::$_10::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/tests/gtests_async_tasks.cpp:99:30 (gtests_dbms+0x5afad66)
[2023-12-28T18:33:49.698Z]     #1 decltype(static_cast<DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody()::$_10&>(fp)()) std::__1::__invoke<DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody()::$_10&>(DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody()::$_10&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0x5afad66)
[2023-12-28T18:33:49.698Z]     #2 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody()::$_10&>(DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody()::$_10&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0x5afad66)
[2023-12-28T18:33:49.698Z]     #3 std::__1::__function::__alloc_func<DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody()::$_10, std::__1::allocator<DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody()::$_10>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0x5afad66)
[2023-12-28T18:33:49.698Z]     #4 std::__1::__function::__func<DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody()::$_10, std::__1::allocator<DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody()::$_10>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0x5afad66)
[2023-12-28T18:33:49.698Z]     #5 std::__1::__function::__value_func<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0x5b04769)
[2023-12-28T18:33:49.698Z]     #6 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0x5b04769)
[2023-12-28T18:33:49.698Z]     #7 DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/Utils/AsyncTasks.h:264:17 (gtests_dbms+0x5b04769)
[2023-12-28T18:33:49.698Z]     #8 decltype(static_cast<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&>(fp)()) std::__1::__invoke<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&>(DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0x5b045dd)
[2023-12-28T18:33:49.698Z]     #9 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&>(DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0x5b045dd)
[2023-12-28T18:33:49.698Z]     #10 std::__1::__function::__alloc_func<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'(), std::__1::allocator<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0x5b045dd)
[2023-12-28T18:33:49.698Z]     #11 std::__1::__function::__func<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'(), std::__1::allocator<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0x5b045dd)
[2023-12-28T18:33:49.698Z]     #12 std::__1::__function::__value_func<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xfb14111)
[2023-12-28T18:33:49.699Z]     #13 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xfb14111)
[2023-12-28T18:33:49.699Z]     #14 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::worker(std::__1::__list_iterator<DB::ThreadFromGlobalPoolImpl<false>, void*>) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0xfb14111)
[2023-12-28T18:33:49.699Z]     #15 bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.699Z]     #16 decltype(static_cast<bool>(fp)()) std::__1::__invoke_constexpr<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&>(bool&&) /usr/local/bin/../include/c++/v1/type_traits:3924:1 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.699Z]     #17 decltype(auto) std::__1::__apply_tuple_impl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(bool&&, std::__1::tuple<>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/tuple:1536:1 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.699Z]     #18 decltype(auto) std::__1::apply<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(bool&&, std::__1::tuple<>&) /usr/local/bin/../include/c++/v1/tuple:1545:1 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.699Z]     #19 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()::operator()() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.h:236:17 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.699Z]     #20 decltype(static_cast<bool>(fp)()) std::__1::__invoke<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()&>(bool&&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.699Z]     #21 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()&>(DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.699Z]     #22 std::__1::__function::__alloc_func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.699Z]     #23 std::__1::__function::__func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0xfb1837c)
[2023-12-28T18:33:49.699Z]     #24 std::__1::__function::__value_func<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xfb1199a)
[2023-12-28T18:33:49.699Z]     #25 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xfb1199a)
[2023-12-28T18:33:49.699Z]     #26 DB::ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0xfb1199a)
[2023-12-28T18:33:49.699Z]     #27 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0xfb153b0)
[2023-12-28T18:33:49.700Z]     #28 decltype(static_cast<void>(fp)()) std::__1::__invoke<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xfb153b0)
[2023-12-28T18:33:49.700Z]     #29 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xfb153b0)
[2023-12-28T18:33:49.700Z]     #30 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xfb153b0)
[2023-12-28T18:33:49.700Z] 
[2023-12-28T18:33:49.700Z]   Previous read of size 1 at 0x7ffdc4b030a0 by main thread:
[2023-12-28T18:33:49.700Z]     #0 DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/tests/gtests_async_tasks.cpp:108:17 (gtests_dbms+0x5af6b91)
[2023-12-28T18:33:49.700Z]     #1 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc41fbd5)
[2023-12-28T18:33:49.700Z]     #2 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc41fbd5)
[2023-12-28T18:33:49.700Z]     #3 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xc3f9775)
[2023-12-28T18:33:49.700Z]     #4 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xc3fafa4)
[2023-12-28T18:33:49.700Z]     #5 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xc3fbc6d)
[2023-12-28T18:33:49.700Z]     #6 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xc40be20)
[2023-12-28T18:33:49.700Z]     #7 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc421075)
[2023-12-28T18:33:49.700Z]     #8 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc421075)
[2023-12-28T18:33:49.700Z]     #9 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xc40b108)
[2023-12-28T18:33:49.700Z]     #10 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x5fd310e)
[2023-12-28T18:33:49.700Z]     #11 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x5fd310e)
[2023-12-28T18:33:49.700Z] 
[2023-12-28T18:33:49.700Z]   Location is stack of main thread.
[2023-12-28T18:33:49.700Z] 
[2023-12-28T18:33:49.700Z]   Location is global '??' at 0x7ffdc4ae5000 ([stack]+0x00000001e0a0)
[2023-12-28T18:33:49.700Z] 
[2023-12-28T18:33:49.700Z]   Thread T105 'ThreadPool' (tid=467165, running) created by main thread at:
[2023-12-28T18:33:49.700Z]     #0 pthread_create /llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:977:3 (gtests_dbms+0x2b9f4fd)
[2023-12-28T18:33:49.700Z]     #1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /usr/local/bin/../include/c++/v1/__threading_support:514:10 (gtests_dbms+0xfb14de8)
[2023-12-28T18:33:49.700Z]     #2 std::__1::thread::thread<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'(), void>(void&&) /usr/local/bin/../include/c++/v1/thread:307:16 (gtests_dbms+0xfb14de8)
[2023-12-28T18:33:49.700Z]     #3 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0xfb10008)
[2023-12-28T18:33:49.700Z]     #4 DB::ThreadPoolImpl<std::__1::thread>::scheduleOrThrow(std::__1::function<void ()>, long, unsigned long, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:198:5 (gtests_dbms+0xfb17ef4)
[2023-12-28T18:33:49.701Z]     #5 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.h:219:38 (gtests_dbms+0xfb17ef4)
[2023-12-28T18:33:49.701Z]     #6 bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0xfb12b80)
[2023-12-28T18:33:49.701Z]     #7 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::trySchedule(std::__1::function<void ()>, long, unsigned long) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/UniThreadPool.cpp:188:12 (gtests_dbms+0xfb1279a)
[2023-12-28T18:33:49.701Z]     #8 DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/Utils/AsyncTasks.h:273:33 (gtests_dbms+0x5b008c8)
[2023-12-28T18:33:49.701Z]     #9 DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTask(unsigned long, std::__1::function<void ()>) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/Utils/AsyncTasks.h:285:16 (gtests_dbms+0x5affa18)
[2023-12-28T18:33:49.701Z]     #10 DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/tests/gtests_async_tasks.cpp:37:22 (gtests_dbms+0x5af5cfa)
[2023-12-28T18:33:49.701Z]     #11 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc41fbd5)
[2023-12-28T18:33:49.701Z]     #12 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc41fbd5)
[2023-12-28T18:33:49.701Z]     #13 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xc3f9775)
[2023-12-28T18:33:49.701Z]     #14 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xc3fafa4)
[2023-12-28T18:33:49.701Z]     #15 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xc3fbc6d)
[2023-12-28T18:33:49.701Z]     #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xc40be20)
[2023-12-28T18:33:49.701Z]     #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc421075)
[2023-12-28T18:33:49.701Z]     #18 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc421075)
[2023-12-28T18:33:49.701Z]     #19 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xc40b108)
[2023-12-28T18:33:49.701Z]     #20 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x5fd310e)
[2023-12-28T18:33:49.701Z]     #21 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x5fd310e)
[2023-12-28T18:33:49.701Z] 
[2023-12-28T18:33:49.701Z] SUMMARY: ThreadSanitizer: data race /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/tests/gtests_async_tasks.cpp:99:30 in DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody()::$_10::operator()() const
[2023-12-28T18:33:49.701Z] ==================
[2023-12-28T18:33:49.701Z] [2023/12/29 02:33:47.110 +08:00] [INFO] [AsyncTasks.h:56] ["Pending 1 tasks when destructing"] [thread_id=1]
[2023-12-28T18:33:49.701Z] [2023/12/29 02:33:47.110 +08:00] [INFO] [gtests_async_tasks.cpp:120] ["Block cancel"] [thread_id=1]
[2023-12-28T18:33:49.701Z] ==================
...
[2023-12-28T18:33:49.732Z] SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/usr/local/lib/x86_64-unknown-linux-gnu/libc++.so.1+0x66f45) in std::__1::mutex::lock()
[2023-12-28T18:33:49.732Z] ==================

4. What is your TiFlash version? (Required)

master
@JaySon-Huang JaySon-Huang added the type/bug The issue is confirmed as a bug. label Dec 29, 2023
@CalvinNeo
Copy link
Member 

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=624544)
  Cycle in lock order graph: M0 (0x7ffd1c3b3308) => M1 (0x7b10000b02d8) => M0

  Mutex M1 acquired here while holding mutex M0 in main thread:
    #0 pthread_mutex_lock /DATA/disk1/qiuyang/projects/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:4457:3 (gtests_dbms+0x2fbd05a)
    #1 std::__1::mutex::lock() <null> (libc++.so.1+0x8fe05)
    #2 DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTask(unsigned long, std::__1::function<void ()>) /DATA/disk1/calvin/tiflash/tics/dbms/src/Storages/KVStore/Utils/AsyncTasks.h:285:16 (gtests_dbms+0x6408c73)
    #3 DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody() /DATA/disk1/calvin/tiflash/tics/dbms/src/Storages/KVStore/tests/gtests_async_tasks.cpp:90:34 (gtests_dbms+0x6406c54)
    #4 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xd3a9929)
    #5 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xd3a9929)
    #6 testing::Test::Run() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xd37d2e2)
    #7 testing::TestInfo::Run() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xd37eb7d)
    #8 testing::TestCase::Run() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xd37f8a6)
    #9 testing::internal::UnitTestImpl::RunAllTests() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xd3923ec)
    #10 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xd3ab049)
    #11 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xd3ab049)
    #12 testing::UnitTest::Run() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xd391c95)
    #13 RUN_ALL_TESTS() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x69788a9)
    #14 main /DATA/disk1/calvin/tiflash/tics/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x69788a9)

  Mutex M0 previously acquired by the same thread here:
    #0 pthread_mutex_lock /DATA/disk1/qiuyang/projects/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:4457:3 (gtests_dbms+0x2fbd05a)
    #1 std::__1::mutex::lock() <null> (libc++.so.1+0x8fe05)
    #2 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xd3a9929)
    #3 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xd3a9929)
    #4 testing::Test::Run() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xd37d2e2)
    #5 testing::TestInfo::Run() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xd37eb7d)
    #6 testing::TestCase::Run() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xd37f8a6)
    #7 testing::internal::UnitTestImpl::RunAllTests() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xd3923ec)
    #8 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xd3ab049)
    #9 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xd3ab049)
    #10 testing::UnitTest::Run() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xd391c95)
    #11 RUN_ALL_TESTS() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x69788a9)
    #12 main /DATA/disk1/calvin/tiflash/tics/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x69788a9)

  Mutex M0 acquired here while holding mutex M1 in thread T81:
    #0 pthread_mutex_lock /DATA/disk1/qiuyang/projects/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:4457:3 (gtests_dbms+0x2fbd05a)
    #1 std::__1::mutex::lock() <null> (libc++.so.1+0x8fe05)
    #2 std::__1::__function::__value_func<void ()>::operator()[abi:v15007]() const /usr/local/bin/../include/c++/v1/__functional/function.h:512:16 (gtests_dbms+0x640bcf4)
    #3 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1197:12 (gtests_dbms+0x640bcf4)
    #4 decltype(std::declval<std::__1::function<void ()>&>()()) std::__1::__invoke[abi:v15007]<std::__1::function<void ()>&>(std::__1::function<void ()>&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:394:23 (gtests_dbms+0x640bcf4)
    #5 std::__1::__packaged_task_func<std::__1::function<void ()>, std::__1::allocator<std::__1::function<void ()>>, void ()>::operator()() /usr/local/bin/../include/c++/v1/future:1694:12 (gtests_dbms+0x640bcf4)
    #6 std::__1::__packaged_task_function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/future:1876:12 (gtests_dbms+0x30cf213)
    #7 std::__1::packaged_task<void ()>::operator()() /usr/local/bin/../include/c++/v1/future:2068:9 (gtests_dbms+0x30cf213)
    #8 DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()::operator()() const /DATA/disk1/calvin/tiflash/tics/dbms/src/Storages/KVStore/Utils/AsyncTasks.h:270:13 (gtests_dbms+0x640c307)
    #9 decltype(std::declval<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&>()()) std::__1::__invoke[abi:v15007]<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&>(DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:394:23 (gtests_dbms+0x640c1bd)
    #10 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&>(DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:479:9 (gtests_dbms+0x640c1bd)
    #11 std::__1::__function::__alloc_func<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'(), std::__1::allocator<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()>, void ()>::operator()[abi:v15007]() /usr/local/bin/../include/c++/v1/__functional/function.h:185:16 (gtests_dbms+0x640c1bd)
    #12 std::__1::__function::__func<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'(), std::__1::allocator<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:359:12 (gtests_dbms+0x640c1bd)
    #13 std::__1::__function::__value_func<void ()>::operator()[abi:v15007]() const /usr/local/bin/../include/c++/v1/__functional/function.h:512:16 (gtests_dbms+0x10e2a2ba)
    #14 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1197:12 (gtests_dbms+0x10e2a2ba)
    #15 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::worker(std::__1::__list_iterator<DB::ThreadFromGlobalPoolImpl<false>, void*>) /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0x10e2a2ba)
    #16 bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0x10e2ec5c)
    #17 decltype(std::declval<bool>()()) std::__1::__invoke[abi:v15007]<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&>(bool&&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:394:23 (gtests_dbms+0x10e2ec5c)
    #18 decltype(auto) std::__1::__apply_tuple_impl[abi:v15007]<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(bool&&, std::__1::tuple<>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/tuple:1789:1 (gtests_dbms+0x10e2ec5c)
    #19 decltype(auto) std::__1::apply[abi:v15007]<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(bool&&, std::__1::tuple<>&) /usr/local/bin/../include/c++/v1/tuple:1798:1 (gtests_dbms+0x10e2ec5c)
    #20 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()::operator()() /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.h:236:17 (gtests_dbms+0x10e2ec5c)
    #21 decltype(std::declval<bool>()()) std::__1::__invoke[abi:v15007]<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()&>(bool&&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:394:23 (gtests_dbms+0x10e2ec5c)
    #22 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()&>(DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:479:9 (gtests_dbms+0x10e2ec5c)
    #23 std::__1::__function::__alloc_func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()>, void ()>::operator()[abi:v15007]() /usr/local/bin/../include/c++/v1/__functional/function.h:185:16 (gtests_dbms+0x10e2ec5c)
    #24 std::__1::__function::__func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:359:12 (gtests_dbms+0x10e2ec5c)
    #25 std::__1::__function::__value_func<void ()>::operator()[abi:v15007]() const /usr/local/bin/../include/c++/v1/__functional/function.h:512:16 (gtests_dbms+0x10e27e13)
    #26 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1197:12 (gtests_dbms+0x10e27e13)
    #27 DB::ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0x10e27e13)
    #28 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0x10e2bb11)
    #29 decltype(std::declval<void>()()) std::__1::__invoke[abi:v15007]<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:394:23 (gtests_dbms+0x10e2bb11)
    #30 void std::__1::__thread_execute[abi:v15007]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:284:5 (gtests_dbms+0x10e2bb11)
    #31 void* std::__1::__thread_proxy[abi:v15007]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) /usr/local/bin/../include/c++/v1/thread:295:5 (gtests_dbms+0x10e2bb11)

  Mutex M1 previously acquired by the same thread here:
    #0 pthread_mutex_lock /DATA/disk1/qiuyang/projects/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:4457:3 (gtests_dbms+0x2fbd05a)
    #1 std::__1::mutex::lock() <null> (libc++.so.1+0x8fe05)
    #2 decltype(std::declval<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&>()()) std::__1::__invoke[abi:v15007]<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&>(DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:394:23 (gtests_dbms+0x640c1bd)
    #3 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&>(DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:479:9 (gtests_dbms+0x640c1bd)
    #4 std::__1::__function::__alloc_func<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'(), std::__1::allocator<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()>, void ()>::operator()[abi:v15007]() /usr/local/bin/../include/c++/v1/__functional/function.h:185:16 (gtests_dbms+0x640c1bd)
    #5 std::__1::__function::__func<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'(), std::__1::allocator<DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>)::'lambda'()>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:359:12 (gtests_dbms+0x640c1bd)
    #6 std::__1::__function::__value_func<void ()>::operator()[abi:v15007]() const /usr/local/bin/../include/c++/v1/__functional/function.h:512:16 (gtests_dbms+0x10e2a2ba)
    #7 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1197:12 (gtests_dbms+0x10e2a2ba)
    #8 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::worker(std::__1::__list_iterator<DB::ThreadFromGlobalPoolImpl<false>, void*>) /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0x10e2a2ba)
    #9 bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0x10e2ec5c)
    #10 decltype(std::declval<bool>()()) std::__1::__invoke[abi:v15007]<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&>(bool&&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:394:23 (gtests_dbms+0x10e2ec5c)
    #11 decltype(auto) std::__1::__apply_tuple_impl[abi:v15007]<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(bool&&, std::__1::tuple<>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/tuple:1789:1 (gtests_dbms+0x10e2ec5c)
    #12 decltype(auto) std::__1::apply[abi:v15007]<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(bool&&, std::__1::tuple<>&) /usr/local/bin/../include/c++/v1/tuple:1798:1 (gtests_dbms+0x10e2ec5c)
    #13 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()::operator()() /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.h:236:17 (gtests_dbms+0x10e2ec5c)
    #14 decltype(std::declval<bool>()()) std::__1::__invoke[abi:v15007]<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()&>(bool&&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:394:23 (gtests_dbms+0x10e2ec5c)
    #15 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()&>(DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:479:9 (gtests_dbms+0x10e2ec5c)
    #16 std::__1::__function::__alloc_func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()>, void ()>::operator()[abi:v15007]() /usr/local/bin/../include/c++/v1/__functional/function.h:185:16 (gtests_dbms+0x10e2ec5c)
    #17 std::__1::__function::__func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:359:12 (gtests_dbms+0x10e2ec5c)
    #18 std::__1::__function::__value_func<void ()>::operator()[abi:v15007]() const /usr/local/bin/../include/c++/v1/__functional/function.h:512:16 (gtests_dbms+0x10e27e13)
    #19 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1197:12 (gtests_dbms+0x10e27e13)
    #20 DB::ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0x10e27e13)
    #21 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0x10e2bb11)
    #22 decltype(std::declval<void>()()) std::__1::__invoke[abi:v15007]<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:394:23 (gtests_dbms+0x10e2bb11)
    #23 void std::__1::__thread_execute[abi:v15007]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:284:5 (gtests_dbms+0x10e2bb11)
    #24 void* std::__1::__thread_proxy[abi:v15007]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) /usr/local/bin/../include/c++/v1/thread:295:5 (gtests_dbms+0x10e2bb11)

  Thread T81 'ThreadPool' (tid=624627, running) created by main thread at:
    #0 pthread_create /DATA/disk1/qiuyang/projects/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1022:3 (gtests_dbms+0x2f9f37d)
    #1 std::__1::__libcpp_thread_create[abi:v15007](unsigned long*, void* (*)(void*), void*) /usr/local/bin/../include/c++/v1/__threading_support:376:10 (gtests_dbms+0x10e2b3ad)
    #2 std::__1::thread::thread<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'(), void>(void&&) /usr/local/bin/../include/c++/v1/thread:311:16 (gtests_dbms+0x10e2b3ad)
    #3 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0x10e266d5)
    #4 DB::ThreadPoolImpl<std::__1::thread>::scheduleOrThrow(std::__1::function<void ()>, long, unsigned long, bool) /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.cpp:198:5 (gtests_dbms+0x10e2e6ed)
    #5 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&) /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.h:219:38 (gtests_dbms+0x10e2e6ed)
    #6 bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0x10e29129)
    #7 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::trySchedule(std::__1::function<void ()>, long, unsigned long) /DATA/disk1/calvin/tiflash/tics/dbms/src/Common/UniThreadPool.cpp:188:12 (gtests_dbms+0x10e28d08)
    #8 DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTaskWithCancel(unsigned long, std::__1::function<void ()>, std::__1::function<void ()>) /DATA/disk1/calvin/tiflash/tics/dbms/src/Storages/KVStore/Utils/AsyncTasks.h:273:33 (gtests_dbms+0x6409703)
    #9 DB::AsyncTasks<unsigned long, std::__1::function<void ()>, void>::addTask(unsigned long, std::__1::function<void ()>) /DATA/disk1/calvin/tiflash/tics/dbms/src/Storages/KVStore/Utils/AsyncTasks.h:285:16 (gtests_dbms+0x6408c73)
    #10 DB::tests::AsyncTasksTest_AsyncTasksNormal_Test::TestBody() /DATA/disk1/calvin/tiflash/tics/dbms/src/Storages/KVStore/tests/gtests_async_tasks.cpp:90:34 (gtests_dbms+0x6406c54)
    #11 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xd3a9929)
    #12 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xd3a9929)
    #13 testing::Test::Run() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xd37d2e2)
    #14 testing::TestInfo::Run() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xd37eb7d)
    #15 testing::TestCase::Run() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xd37f8a6)
    #16 testing::internal::UnitTestImpl::RunAllTests() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xd3923ec)
    #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xd3ab049)
    #18 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xd3ab049)
    #19 testing::UnitTest::Run() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xd391c95)
    #20 RUN_ALL_TESTS() /DATA/disk1/calvin/tiflash/tics/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x69788a9)
    #21 main /DATA/disk1/calvin/tiflash/tics/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x69788a9)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/usr/local/lib/x86_64-unknown-linux-gnu/libc++.so.1+0x8fe05) in std::__1::mutex::lock()

@CalvinNeo CalvinNeo mentioned this issue Jan 2, 2024
Merged
12 tasks
@ti-chi-bot ti-chi-bot bot closed this as completed in #8614 Jan 3, 2024
ti-chi-bot bot pushed a commit that referenced this issue Jan 3, 2024
@CalvinNeo
KVStore: introduce flexible region format (#8614)
7273bbc 
close #8590, close #8624, close #8625
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CalvinNeo CalvinNeo

Labels
component/storage severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

KVStore: introduce flexible region format CalvinNeo/tiflash
2 participants
@JaySon-Huang @CalvinNeo