TiFlash PageStorage V3 crash after restart

[hehechen]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Enable PageStorage V3 and encryption, run CH-benCHmark 1500 warehouses, after two OOM, TiFlash throw exception when restart.

2. What did you expect to see? (Required)

TiFlash successfully restart.

3. What did you see instead (Required)

TiFlash throw exception when restart.

[2022/05/07 17:13:04.294 +08:00] [ERROR] [] ["Application:DB::Exception: Something wrong while reading log file"] [thread_id=1]
[2022/05/07 17:15:19.203 +08:00] [ERROR] [Exception.cpp:85] ["bool DB::Context::initializeGlobalStoragePoolIfNeed(const DB::PathPool &, bool):Code: 0, e.displayText() = DB::Exception: [header_off=11] [payload_length=12662] [checksum=0x3A70CB628F571380] [actual_checksum=0xDCC2066722BB4097] [file=/data2/chentongli/tidb-data/tiflash-19111/page/log/wal/log_5_1], e.what() = DB::Exception, Stack trace:\n\n\n 0x4446e8e\tStackTrace::StackTrace() [tiflash+71593614]\n \tdbms/src/Common/StackTrace.cpp:23\n 0x44440d2\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) [tiflash+71581906]\n \tdbms/src/Common/Exception.h:41\n 0xdae76ae\tDB::PS::V3::LogReader::readPhysicalRecord(std::__1::basic_string_view<char, std::__1::char_traits >, unsigned long) [tiflash+229537454]\n \tdbms/src/Storages/Page/V3/LogFile/LogReader.cpp:414\n 0xdae6601\tDB::PS::V3::LogReader::readRecord() [tiflash+229533185]\n \tdbms/src/Storages/Page/V3/LogFile/LogReader.cpp:122\n 0xdadceef\tDB::PS::V3::WALStoreReader::next() [tiflash+229494511]\n \tdbms/src/Storages/Page/V3/WAL/WALReader.cpp:179\n 0xdaa76bf\tDB::PS::V3::PageDirectoryFactory::loadFromDisk(std::__1::unique_ptr<DB::PS::V3::PageDirectory, std::__1::default_deleteDB::PS::V3::PageDirectory > const&, std::__1::shared_ptrDB::PS::V3::WALStoreReader&&) [tiflash+229275327]\n \tdbms/src/Storages/Page/V3/PageDirectoryFactory.cpp:175\n 0xdaa73c8\tDB::PS::V3::PageDirectoryFactory::create(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::shared_ptrDB::FileProvider&, std::__1::shared_ptrDB::PSDiskDelegator&, DB::PS::V3::WALStore::Config) [tiflash+229274568]\n \tdbms/src/Storages/Page/V3/PageDirectoryFactory.cpp:29\n 0xda5c248\tDB::PS::V3::PageStorageImpl::restore() [tiflash+228966984]\n \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:55\n 0xd625578\tDB::DM::GlobalStoragePool::restore() [tiflash+224548216]\n \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:116\n 0xcad9212\tDB::Context::initializeGlobalStoragePoolIfNeed(DB::PathPool const&, bool) [tiflash+212701714]\n \tdbms/src/Interpreters/Context.cpp:1595\n 0x448b93b\tDB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > > const&) [tiflash+71874875]\n \tdbms/src/Server/Server.cpp:1120\n 0xe380f15\tPoco::Util::Application::run() [tiflash+238554901]\n \tcontrib/poco/Util/src/Application.cpp:335\n 0xe39bb25\tPoco::Util::ServerApplication::run() [tiflash+238664485]\n \tcontrib/poco/Util/src/ServerApplication.cpp:94\n 0xe39bd03\tPoco::Util::ServerApplication::run(int, char**) [tiflash+238664963]\n \tcontrib/poco/Util/src/ServerApplication.cpp:618\n 0x4495228\tmainEntryClickHouseServer(int, char**) [tiflash+71914024]\n \tdbms/src/Server/Server.cpp:1545\n 0x4414f9f\tmain [tiflash+71389087]\n \tdbms/src/Server/main.cpp:167\n 0x7f4c0a9d7555\t__libc_start_main [libc.so.6+140629]\n 0x4414969\t [tiflash+71387497]\n \t/root/yuzhao/glibc-2.17/csu/../sysdeps/x86_64/start.S:123"] [thread_id=1]
[2022/05/07 17:15:19.300 +08:00] [ERROR] [] ["Application:DB::Exception: [header_off=11] [payload_length=12662] [checksum=0x3A70CB628F571380] [actual_checksum=0xDCC2066722BB4097] [file=/data2/chentongli/tidb-data/tiflash-19111/page/log/wal/log_5_1]"] [thread_id=1]
[2022/05/07 17:31:02.816 +08:00] [ERROR] [Exception.cpp:85] ["bool DB::Context::initializeGlobalStoragePoolIfNeed(const DB::PathPool &, bool):Code: 0, e.displayText() = DB::Exception: [header_off=11] [payload_length=1657] [checksum=0x9BE2C0FA71209210] [actual_checksum=0x9D0DD04ACBC4BB34] [file=/data2/chentongli/tidb-data/tiflash-19111/page/meta/wal/log_5_1], e.what() = DB::Exception, Stack trace:\n\n\n 0x4446e8e\tStackTrace::StackTrace() [tiflash+71593614]\n \tdbms/src/Common/StackTrace.cpp:23\n 0x44440d2\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) [tiflash+71581906]\n \tdbms/src/Common/Exception.h:41\n 0xdae76ae\tDB::PS::V3::LogReader::readPhysicalRecord(std::__1::basic_string_view<char, std::__1::char_traits >, unsigned long) [tiflash+229537454]\n \tdbms/src/Storages/Page/V3/LogFile/LogReader.cpp:414\n 0xdae6601\tDB::PS::V3::LogReader::readRecord() [tiflash+229533185]\n \tdbms/src/Storages/Page/V3/LogFile/LogReader.cpp:122\n 0xdadceef\tDB::PS::V3::WALStoreReader::next() [tiflash+229494511]\n \tdbms/src/Storages/Page/V3/WAL/WALReader.cpp:179\n 0xdaa76bf\tDB::PS::V3::PageDirectoryFactory::loadFromDisk(std::__1::unique_ptr<DB::PS::V3::PageDirectory, std::__1::default_deleteDB::PS::V3::PageDirectory > const&, std::__1::shared_ptrDB::PS::V3::WALStoreReader&&) [tiflash+229275327]\n \tdbms/src/Storages/Page/V3/PageDirectoryFactory.cpp:175\n 0xdaa73c8\tDB::PS::V3::PageDirectoryFactory::create(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::shared_ptrDB::FileProvider&, std::__1::shared_ptrDB::PSDiskDelegator&, DB::PS::V3::WALStore::Config) [tiflash+229274568]\n \tdbms/src/Storages/Page/V3/PageDirectoryFactory.cpp:29\n 0xda5c248\tDB::PS::V3::PageStorageImpl::restore() [tiflash+228966984]\n \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:55\n 0xd62557c\tDB::DM::GlobalStoragePool::restore() [tiflash+224548220]\n \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:116\n 0xcad9212\tDB::Context::initializeGlobalStoragePoolIfNeed(DB::PathPool const&, bool) [tiflash+212701714]\n \tdbms/src/Interpreters/Context.cpp:1595\n 0x448b93b\tDB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > > const&) [tiflash+71874875]\n \tdbms/src/Server/Server.cpp:1120\n 0xe380f15\tPoco::Util::Application::run() [tiflash+238554901]\n \tcontrib/poco/Util/src/Application.cpp:335\n 0xe39bb25\tPoco::Util::ServerApplication::run() [tiflash+238664485]\n \tcontrib/poco/Util/src/ServerApplication.cpp:94\n 0xe39bd03\tPoco::Util::ServerApplication::run(int, char**) [tiflash+238664963]\n \tcontrib/poco/Util/src/ServerApplication.cpp:618\n 0x4495228\tmainEntryClickHouseServer(int, char**) [tiflash+71914024]\n \tdbms/src/Server/Server.cpp:1545\n 0x4414f9f\tmain [tiflash+71389087]\n \tdbms/src/Server/main.cpp:167\n 0x7f6343fab555\t__libc_start_main [libc.so.6+140629]\n 0x4414969\t [tiflash+71387497]\n \t/root/yuzhao/glibc-2.17/csu/../sysdeps/x86_64/start.S:123"] [thread_id=1]
[2022/05/07 17:31:02.989 +08:00] [ERROR] [] ["Application:DB::Exception: [header_off=11] [payload_length=1657] [checksum=0x9BE2C0FA71209210] [actual_checksum=0x9D0DD04ACBC4BB34] [file=/data2/chentongli/tidb-data/tiflash-19111/page/meta/wal/log_5_1]"] [thread_id=1]

4. What is your TiFlash version? (Required)

04da47f