Disaggregated TiFlash crash when the process is shutting down

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

stale read and chaos

[2024/03/11 20:29:49.078 +08:00] [ERROR] [BaseDaemon.cpp:358] ["(from thread 48) Terminate called without an active exception
"] [source=BaseDaemon] [thread_id=552]
[2024/03/11 20:29:49.078 +08:00] [ERROR] [BaseDaemon.cpp:367] [########################################] [source=BaseDaemon] [thread_id=552]
[2024/03/11 20:29:49.078 +08:00] [ERROR] [BaseDaemon.cpp:368] ["(from thread 48) Received signal Aborted(6)."] [source=BaseDaemon] [thread_id=552]
[2024/03/11 20:29:49.080 +08:00] [ERROR] [BaseDaemon.cpp:560] ["
       0x77c17c1    faultSignalHandler(int, siginfo_t*, void*) [tiflash+125573057]
                    libs/libdaemon/src/BaseDaemon.cpp:211
  0x7fe0caf9fdb0    <unknown symbol> [libc.so.6+347568]
  0x7fe0cafec54c    __pthread_kill_implementation [libc.so.6+660812]
  0x7fe0caf9fd06    __GI_raise [libc.so.6+347398]
  0x7fe0caf737f3    abort [libc.so.6+165875]
       0x77c0eb4    terminate_handler() [tiflash+125570740]
                    libs/libdaemon/src/BaseDaemon.cpp:0
  0x7fe0cfbe4a13    std::__terminate(void (*)()) [libc++abi.so.1+236051]
  0x7fe0cfbe49a2    std::terminate() [libc++abi.so.1+235938]
  0x7fe0cfc5f8ce    std::__1::thread::~thread() [libc++.so.1+461006]
       0x809999c    DB::BackgroundProcessingPool::~BackgroundProcessingPool() [tiflash+134846876]
                    dbms/src/Storages/BackgroundProcessingPool.cpp:151
       0x874b644    DB::UniversalPageStorageService::~UniversalPageStorageService() [tiflash+141866564]
                    dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp:297
       0x874b857    std::__1::__shared_ptr_pointer<DB::UniversalPageStorageService*, std::__1::shared_ptr<DB::UniversalPageStorageService>::__shared_ptr_default_delete<DB::UniversalPageStorageService, DB::UniversalPageStorageService>, std::__1::allocator<DB::UniversalPageStorageService> >::__on_zero_shared() [tiflash+141867095]
                    /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:267
       0x874b9e1    std::__1::__function::__func<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_3, std::__1::allocator<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_3>, bool ()>::destroy() [tiflash+141867489]
                    /usr/local/bin/../include/c++/v1/__functional/function.h:327
       0x809ad6c    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1> >(void*) [tiflash+134851948]
                    /usr/local/bin/../include/c++/v1/thread:291
  0x7fe0cafea802    start_thread [libc.so.6+653314]"] [source=BaseDaemon] [thread_id=552]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

master ee471e9

[JaySon-Huang]

the shutdown is cause by chaos kill pod in the test env

[2024/03/11 20:29:30.064 +08:00] [INFO] [BaseDaemon.cpp:1313] ["Received termination signal (Terminated)"] [source=Application] [thread_id=552]
...
[2024/03/11 20:29:41.345 +08:00] [INFO] [Server.cpp:713] ["Received termination signal, stopping server..."] [thread_id=1]
[2024/03/11 20:29:48.159 +08:00] [DEBUG] [Server.cpp:1536] ["Shutted down storages."] [thread_id=1]
[2024/03/11 20:29:48.898 +08:00] [INFO] [AsyncTasks.h:56] ["Pending 0 tasks when destructing"] [thread_id=1]
[2024/03/11 20:29:49.078 +08:00] [ERROR] [Exception.cpp:96] ["std::exception. Code: 1001, type: std::__1::system_error, e.what() = thread::join failed: Resource deadlock avoided"] [source=DB::BackgroundProcessingPool::~BackgroundProcessingPool()] [thread_id=48]
[2024/03/11 20:29:49.078 +08:00] [ERROR] [BaseDaemon.cpp:358] ["(from thread 48) Terminate called without an active exception\n"] [source=BaseDaemon] [thread_id=552]
[2024/03/11 20:29:49.078 +08:00] [ERROR] [BaseDaemon.cpp:367] [########################################] [source=BaseDaemon] [thread_id=552]
[2024/03/11 20:29:49.078 +08:00] [ERROR] [BaseDaemon.cpp:368] ["(from thread 48) Received signal Aborted(6)."] [source=BaseDaemon] [thread_id=552]
[2024/03/11 20:29:49.080 +08:00] [ERROR] [BaseDaemon.cpp:560] ["
       0x77c17c1    faultSignalHandler(int, siginfo_t*, void*) [tiflash+125573057]
                    libs/libdaemon/src/BaseDaemon.cpp:211
  0x7fe0caf9fdb0    <unknown symbol> [libc.so.6+347568]
  0x7fe0cafec54c    __pthread_kill_implementation [libc.so.6+660812]
  0x7fe0caf9fd06    __GI_raise [libc.so.6+347398]
  0x7fe0caf737f3    abort [libc.so.6+165875]
       0x77c0eb4    terminate_handler() [tiflash+125570740]
                    libs/libdaemon/src/BaseDaemon.cpp:0
  0x7fe0cfbe4a13    std::__terminate(void (*)()) [libc++abi.so.1+236051]
  0x7fe0cfbe49a2    std::terminate() [libc++abi.so.1+235938]
  0x7fe0cfc5f8ce    std::__1::thread::~thread() [libc++.so.1+461006]
       0x809999c    DB::BackgroundProcessingPool::~BackgroundProcessingPool() [tiflash+134846876]
                    dbms/src/Storages/BackgroundProcessingPool.cpp:151
       0x874b644    DB::UniversalPageStorageService::~UniversalPageStorageService() [tiflash+141866564]
                    dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp:297
       0x874b857    std::__1::__shared_ptr_pointer<DB::UniversalPageStorageService*, std::__1::shared_ptr<DB::UniversalPageStorageService>::__shared_ptr_default_delete<DB::UniversalPageStorageService, DB::UniversalPageStorageService>, std::__1::allocator<DB::UniversalPageStorageService> >::__on_zero_shared() [tiflash+141867095]
                    /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:267
       0x874b9e1    std::__1::__function::__func<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_3, std::__1::allocator<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_3>, bool ()>::destroy() [tiflash+141867489]
                    /usr/local/bin/../include/c++/v1/__functional/function.h:327
       0x809ad6c    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1> >(void*) [tiflash+134851948]
                    /usr/local/bin/../include/c++/v1/thread:291
  0x7fe0cafea802    start_thread [libc.so.6+653314]"] [source=BaseDaemon] [thread_id=552]

[2024/03/11 20:29:49.778 +08:00] [INFO] [KVStore.cpp:427] ["Destroy KVStore"] [thread_id=1]