EstablishDisaggTask meet exception: DB::Exception: Logical error: Sizes of nested column and null map of Nullable column are not equal

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

deploy tiflash with S3 disagg mode

2. What did you expect to see? (Required)

3. What did you see instead (Required)

[2023/03/24 13:16:21.173 +08:00] [DEBUG] [WNEstablishDisaggTaskHandler.cpp:50] ["DisaggregatedTask handling 1 regions from 1 physical tables"] [source="DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19>"] [thread_id=96]
[2023/03/24 13:16:21.173 +08:00] [DEBUG] [WNEstablishDisaggTaskHandler.cpp:69] ["DAGReq: time_zone_offset: 28800 flags: 226 output_offsets: 0 output_offsets: 1 output_offsets: 2 output_offsets: 3 output_offsets: 4 output_offsets: 5 output_offsets: 6 encode_type: TypeCHBlock time_zone_name: \"Asia/Shanghai\" collect_execution_summaries: false root_executor { tp:
TypeTableScan tbl_scan { table_id: 111 columns { column_id: 1 tp: 3 collation: -63 columnLen: 11 decimal: 0 flag: 4099 pk_handle: false } columns { column_id: 2 tp: 3 collation: -63 columnLen: 11 decimal: 0 flag: 4099 pk_handle: false } columns { column_id: 3 tp: 3 collation: -63 columnLen: 11 decimal: 0 flag: 4107 pk_handle: false } columns { column_id: 4 tp: 1
5 collation: -46 columnLen: 16 decimal: 0 flag: 0 pk_handle: false } columns { column_id: 5 tp: 254 collation: -46 columnLen: 2 decimal: 0 flag: 0 pk_handle: false } columns { column_id: 6 tp: 15 collation: -46 columnLen: 16 decimal: 0 flag: 0 pk_handle: false } columns { column_id: 17 tp: 246 collation: -63 columnLen: 12 decimal: 2 flag: 0 pk_handle: false } de
sc: false primary_column_ids: 3 primary_column_ids: 2 primary_column_ids: 1 next_read_engine: Local keep_order: false is_fast_scan: false } executor_id: \"TableRangeScan_19\" } force_encode_type: true"] [source="DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRange
Scan_19>"] [thread_id=96]
[2023/03/24 13:16:21.173 +08:00] [DEBUG] [executeQuery.cpp:386] ["(from 172.16.5.85:55258, query_id: 15b6057f-f0d9-4890-91ad-2a165a3444e9) time_zone_offset: 28800 flags: 226 output_offsets: 0 output_offsets: 1 output_offsets: 2 output_offsets: 3 output_offsets: 4 output_offsets: 5 output_offsets: 6 encode_type: TypeCHBlock time_zone_name: \"Asia/Shanghai\" colle
ct_execution_summaries: false root_executor {   tp: TypeTableScan   tbl_scan {     table_id: 111     columns {       column_id: 1       tp: 3       collation: -63       columnLen: 11       decimal: 0       flag: 4099       pk_handle: false     }     columns {       column_id: 2       tp: 3       collation: -63       columnLen: 11       decimal: 0       flag: 409
9       pk_handle: false     }     columns {       column_id: 3       tp: 3       collation: -63       columnLen: 11       decimal: 0       flag: 4107       pk_handle: false     }     columns {       column_id: 4       tp: 15       collation: -46       columnLen: 16       decimal: 0       flag: 0       pk_handle: false     }     columns {       column_id: 5
  tp: 254       collation: -46       columnLen: 2       decimal: 0       flag: 0       pk_handle: false     }     columns {       column_id: 6       tp: 15       collation: -46       columnLen: 16       decimal: 0       flag: 0       pk_handle: false     }     columns {       column_id: 17       tp: 246       collation: -63       columnLen: 12       decimal: 2
     flag: 0       pk_handle: false     }     desc: false     primary_column_ids: 3     primary_column_ids: 2     primary_column_ids: 1     next_read_engine: Local     keep_order: false     is_fast_scan: false   }   executor_id: \"TableRangeScan_19\" } force_encode_type: true "] [source="DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, s
erver_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19>"] [thread_id=96]
[2023/03/24 13:16:21.173 +08:00] [DEBUG] [DeltaValueSpace.cpp:340] ["Update index done, delta=<delta_id=172>"] [source="keyspace_id=1 table_id=121 segment_id=171 epoch=11"] [thread_id=73]
[2023/03/24 13:16:21.173 +08:00] [DEBUG] [PhysicalPlan.cpp:253] ["build unoptimized physical plan: \n<Projection, TableRangeScan_19> | is_tidb_operator: false, schema: <TableRangeScan_19_table_scan_0, Int32>, <TableRangeScan_19_table_scan_1, Int32>, <TableRangeScan_19_table_scan_2, Int32>, <TableRangeScan_19_table_scan_3, Nullable(String)>, <TableRangeScan_19_ta
ble_scan_4, Nullable(String)>, <TableRangeScan_19_table_scan_5, Nullable(String)>, <TableRangeScan_19_table_scan_6, Nullable(Decimal(12,2))>\n <TableScan, TableRangeScan_19> | is_tidb_operator: true, schema: <table_scan_0, Int32>, <table_scan_1, Int32>, <table_scan_2, Int32>, <table_scan_3, Nullable(String)>, <table_scan_4, Nullable(String)>, <table_scan_5, Null
able(String)>, <table_scan_6, Nullable(Decimal(12,2))>\n"] [source="DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19>"] [thread_id=96]
[2023/03/24 13:16:21.174 +08:00] [DEBUG] [PhysicalPlan.cpp:259] ["build optimized physical plan: \n<Projection, TableRangeScan_19> | is_tidb_operator: false, schema: <TableRangeScan_19_table_scan_0, Int32>, <TableRangeScan_19_table_scan_1, Int32>, <TableRangeScan_19_table_scan_2, Int32>, <TableRangeScan_19_table_scan_3, Nullable(String)>, <TableRangeScan_19_tabl
e_scan_4, Nullable(String)>, <TableRangeScan_19_table_scan_5, Nullable(String)>, <TableRangeScan_19_table_scan_6, Nullable(Decimal(12,2))>\n <TableScan, TableRangeScan_19> | is_tidb_operator: true, schema: <table_scan_0, Int32>, <table_scan_1, Int32>, <table_scan_2, Int32>, <table_scan_3, Nullable(String)>, <table_scan_4, Nullable(String)>, <table_scan_5, Nullab
le(String)>, <table_scan_6, Nullable(Decimal(12,2))>\n"] [source="DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19>"] [thread_id=96]
[2023/03/24 13:16:21.174 +08:00] [DEBUG] [LearnerRead.cpp:280] ["Batch read index, original size 1, send & get 1 message, cost 0ms"] [source="DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19>"] [thread_id=96]
[2023/03/24 13:16:21.174 +08:00] [DEBUG] [Region.cpp:538] ["[region 620, applied: term 7 index 80684] need to wait learner index 80685"] [thread_id=96]
[2023/03/24 13:16:21.174 +08:00] [DEBUG] [Region.cpp:548] ["[region 620] wait learner index 80685 done"] [thread_id=96]
[2023/03/24 13:16:21.174 +08:00] [DEBUG] [LearnerRead.cpp:402] ["Finish wait index | resolve locks | check memory cache for 1 regions, cost 0ms, 0 unavailable regions"] [source="DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19>"] [thread_id=96]
[2023/03/24 13:16:21.174 +08:00] [DEBUG] [LearnerRead.cpp:424] ["[Learner Read] batch read index | wait index total_cost=0 read_cost=0 wait_cost=0 n_regions=1 n_stale_read=0 n_unavailable=0"] [source="DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19>"]
[thread_id=96]
[2023/03/24 13:16:21.174 +08:00] [INFO] [DAGStorageInterpreter.cpp:958] ["Table 111 schema OK, no syncing required. Schema version [storage, global, query]: [63, 81, 81]"] [source="DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19>"] [thread_id=96]
[2023/03/24 13:16:21.174 +08:00] [DEBUG] [StorageDeltaMerge.cpp:651] ["Read with tso: 440306232488886404"] [source="ks_1_db_105.ks_1_t_111 DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19> table_id=111"] [thread_id=96]
[2023/03/24 13:16:21.174 +08:00] [DEBUG] [DeltaMergeStore.cpp:1735] ["[sorted_ranges: 1] [tasks before split: 1] [tasks final: 1] [ranges final: 1]"] [source="keyspace_id=1 table_id=111 DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19> table_id=111"] [t
hread_id=96]
[2023/03/24 13:16:21.174 +08:00] [DEBUG] [DeltaMergeStore.cpp:1180] ["Read create segment snapshot done"] [source="keyspace_id=1 table_id=111 DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19> table_id=111"] [thread_id=96]
[2023/03/24 13:16:21.174 +08:00] [DEBUG] [DAGStorageInterpreter.cpp:796] ["local streams built, is_disaggregated_task=true snap_id=DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19>"] [source="DisTaskId<MPP<query:<query_ts:1679634981160734961, local_quer
y_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19>"] [thread_id=96]
[2023/03/24 13:16:21.174 +08:00] [DEBUG] [WNDisaggSnapshotManager.h:55] ["Register Disaggregated Snapshot, task_id=DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19>"] [thread_id=96]
[2023/03/24 13:16:21.175 +08:00] [DEBUG] [executeQuery.cpp:427] ["Query pipeline:\nExpression: <final projection>\n Expression: <table scan schema projection>\n  Null\n"] [source="DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19>"] [thread_id=96]
[2023/03/24 13:16:21.353 +08:00] [ERROR] [FlashService.cpp:695] ["EstablishDisaggTask meet exception: DB::Exception: Logical error: Sizes of nested column and null map of Nullable column are not equal: null size is : 41 column size is : 31
       0x1b51783    StackTrace::StackTrace() [tiflash+28645251]
                    dbms/src/Common/StackTrace.cpp:23
       0x1b4f586 DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+28636550]
                    dbms/src/Common/Exception.h:46
       0x7c347c0    DB::ColumnNullable::checkConsistency() const [tiflash+130238400]
                    dbms/src/Columns/ColumnNullable.cpp:589
       0x74cd192    DB::DataTypeNullable::serializeBinaryBulkWithMultipleStreams(DB::IColumn const&, std::__1::function<DB::WriteBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, unsigned long, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&) const [tiflash+122474898]
                    dbms/src/DataTypes/DataTypeNullable.cpp:71
       0x72a4201    DB::DM::serializeColumn(DB::WriteBuffer&, DB::IColumn const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long, unsigned long, DB::CompressionMethod, long) [tiflash+120209921]
                    dbms/src/Storages/DeltaMerge/ColumnFile/ColumnFilePersisted.cpp:70
       0x8055033    DB::DM::Remote::Serializer::serializeTo(DB::DM::ColumnFileInMemory const&) [tiflash+134565939]
                    dbms/src/Storages/DeltaMerge/Remote/Serializer.cpp:266
       0x80535be    DB::DM::Remote::Serializer::serializeTo(std::__1::shared_ptr<DB::DM::ColumnFileSetSnapshot> const&) [tiflash+134559166]
                    dbms/src/Storages/DeltaMerge/Remote/Serializer.cpp:183
       0x8052dbe    DB::DM::Remote::Serializer::serializeTo(std::__1::shared_ptr<DB::DM::SegmentSnapshot> const&, unsigned long, unsigned long, DB::DM::RowKeyRange const&, std::__1::vector<DB::DM::RowKeyRange, std::__1::allocator<DB::DM::RowKeyRange> > const&) [tiflash+134557118]
                    dbms/src/Storages/DeltaMerge/Remote/Serializer.cpp:96
       0x8052963    DB::DM::Remote::Serializer::serializeTo(std::__1::unique_ptr<DB::DM::Remote::DisaggPhysicalTableReadSnapshot, std::__1::default_delete<DB::DM::Remote::DisaggPhysicalTableReadSnapshot> > const&, DB::DM::DisaggTaskId const&) [tiflash+134556003]
                    dbms/src/Storages/DeltaMerge/Remote/Serializer.cpp:56
       0x822bbb2    std::__1::__function::__func<DB::WNEstablishDisaggTaskHandler::execute(disaggregated::EstablishDisaggTaskResponse*)::$_4, std::__1::allocator<DB::WNEstablishDisaggTaskHandler::execute(disaggregated::EstablishDisaggTaskResponse*)::$_4>, void (std::__1::unique_ptr<DB::DM::Remote::DisaggPhysicalTableReadSnapshot, std::__1::default_delete<DB::DM::Remote::DisaggPhysicalTableReadSnapshot> > const&)>::operator()(std::__1::unique_ptr<DB::DM::Remote::DisaggPhysicalTableReadSnapshot, std::__1::default_delete<DB::DM::Remote::DisaggPhysicalTableReadSnapshot> > const&) [tiflash+136494002]
                    /data3/jaysonhuang/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345
       0x72f1043    DB::DM::Remote::DisaggReadSnapshot::iterateTableSnapshots(std::__1::function<void (std::__1::unique_ptr<DB::DM::Remote::DisaggPhysicalTableReadSnapshot, std::__1::default_delete<DB::DM::Remote::DisaggPhysicalTableReadSnapshot> > const&)>) const [tiflash+120524867]
                    dbms/src/Storages/DeltaMerge/Remote/DisaggSnapshot.cpp:58
       0x822b8ac    DB::WNEstablishDisaggTaskHandler::execute(disaggregated::EstablishDisaggTaskResponse*) [tiflash+136493228]
                    dbms/src/Flash/Disaggregated/WNEstablishDisaggTaskHandler.cpp:104
       0x7f850ef    DB::FlashService::EstablishDisaggTask(grpc::ServerContext*, disaggregated::EstablishDisaggTaskRequest const*, disaggregated::EstablishDisaggTaskResponse*) [tiflash+133714159]
                    dbms/src/Flash/FlashService.cpp:663
       0x9049d2e    std::__1::__function::__func<tikvpb::Tikv::Service::Service()::$_60, std::__1::allocator<tikvpb::Tikv::Service::Service()::$_60>, grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, disaggregated::EstablishDisaggTaskRequest const*, disaggregated::EstablishDisaggTaskResponse*)>::operator()(tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, disaggregated::EstablishDisaggTaskRequest const*&&, disaggregated::EstablishDisaggTaskResponse*&&) [tiflash+151297326]
                    /data3/jaysonhuang/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345
       0x90923c5    grpc::Status grpc::internal::CatchingFunctionHandler<grpc::internal::RpcMethodHandler<tikvpb::Tikv::Service, disaggregated::EstablishDisaggTaskRequest, disaggregated::EstablishDisaggTaskResponse, google::protobuf::MessageLite, google::protobuf::MessageLite>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()>(grpc::internal::RpcMethodHandler<tikvpb::Tikv::Service, disaggregated::EstablishDisaggTaskRequest, disaggregated::EstablishDisaggTaskResponse, google::protobuf::MessageLite, google::protobuf::MessageLite>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()&&) [tiflash+151593925]
                    contrib/grpc/include/grpcpp/impl/codegen/method_handler.h:44
       0x90921e9    grpc::internal::RpcMethodHandler<tikvpb::Tikv::Service, disaggregated::EstablishDisaggTaskRequest, disaggregated::EstablishDisaggTaskResponse, google::protobuf::MessageLite, google::protobuf::MessageLite>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&) [tiflash+151593449]
                    contrib/grpc/include/grpcpp/impl/codegen/method_handler.h:113
       0x8a88f39    grpc::Server::SyncRequest::ContinueRunAfterInterception() [tiflash+145264441]
                    contrib/grpc/src/cpp/server/server_cc.cc:433
       0x8a88d48    grpc::Server::SyncRequest::Run(std::__1::shared_ptr<grpc::Server::GlobalCallbacks> const&, bool) [tiflash+145263944]
                    contrib/grpc/src/cpp/server/server_cc.cc:421
       0x8a988e7    grpc::ThreadManager::MainWorkLoop() [tiflash+145328359]
                    contrib/grpc/src/cpp/thread_manager/thread_manager.cc:211
       0x8a991b1    grpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::__invoke(void*) [tiflash+145330609]
                    contrib/grpc/src/cpp/thread_manager/thread_manager.cc:36
       0x8e2d2fc    grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) [tiflash+149082876]
                    contrib/grpc/src/core/lib/gprpp/thd_posix.cc:110
  0x7f7cc3441ea5    start_thread [libpthread.so.0+32421]
  0x7f7cc2d4fb0d    clone [libc.so.6+1043213]"] [source="DisTaskId<MPP<query:<query_ts:1679634981160734961, local_query_id:152855, server_id:4080304, start_ts:440306232488886404>,task_id:1>,executor=TableRangeScan_19>"] [thread_id=96]

4. What is your TiFlash version? (Required)

master

[JaySon-Huang]

ref #6882

[JaySon-Huang]

[2023/04/06 13:39:45.994 +00:00] [ERROR] [FlashService.cpp:685] ["EstablishDisaggTask meet exception: DB::Exception: Logical error: Sizes of nested column and null map of Nullable column are not equal: null size is : 1863 column size is : 1854\n\n       0x1b6f443\tStackTrace::StackTrace() [tiflash+28767299]\n                \tdbms/src/Common/StackTrace.cpp:23\n
      0x1b6d246\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+28758598]\n                \tdbms/src/Common/Exception.h:46\n       0x7c8ded0\tDB::ColumnNullable::checkConsistency() const [tiflash+130604752]\n                \tdbms/src/Columns/ColumnNullable.cpp:589\n       0x7
521aa2\tDB::DataTypeNullable::serializeBinaryBulkWithMultipleStreams(DB::IColumn const&, std::__1::function<DB::WriteBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&, unsigned long, unsigned long, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&) c
onst [tiflash+122821282]\n                \tdbms/src/DataTypes/DataTypeNullable.cpp:71\n       0x72f3a11\tDB::DM::serializeColumn(DB::WriteBuffer&, DB::IColumn const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long, unsigned long, DB::CompressionMethod, long) [tiflash+120535569]\n                \tdbms/src/Storages/DeltaMerge/ColumnFile/ColumnFi
lePersisted.cpp:70\n       0x80d2ba3\tDB::DM::Remote::Serializer::serializeTo(DB::DM::ColumnFileInMemory const&) [tiflash+135080867]\n                \tdbms/src/Storages/DeltaMerge/Remote/Serializer.cpp:266\n       0x80d112e\tDB::DM::Remote::Serializer::serializeTo(std::__1::shared_ptr<DB::DM::ColumnFileSetSnapshot> const&) [tiflash+135074094]\n                \
tdbms/src/Storages/DeltaMerge/Remote/Serializer.cpp:183\n       0x80d092e\tDB::DM::Remote::Serializer::serializeTo(std::__1::shared_ptr<DB::DM::SegmentSnapshot> const&, unsigned long, unsigned long, DB::DM::RowKeyRange const&, std::__1::vector<DB::DM::RowKeyRange, std::__1::allocator<DB::DM::RowKeyRange> > const&) [tiflash+135072046]\n                \tdbms/src/
Storages/DeltaMerge/Remote/Serializer.cpp:96\n       0x80d04d3\tDB::DM::Remote::Serializer::serializeTo(std::__1::unique_ptr<DB::DM::Remote::DisaggPhysicalTableReadSnapshot, std::__1::default_delete<DB::DM::Remote::DisaggPhysicalTableReadSnapshot> > const&, DB::DM::DisaggTaskId const&) [tiflash+135070931]\n                \tdbms/src/Storages/DeltaMerge/Remote/Se
rializer.cpp:56\n       0x82ba4a2\tstd::__1::__function::__func<DB::WNEstablishDisaggTaskHandler::execute(disaggregated::EstablishDisaggTaskResponse*)::$_4, std::__1::allocator<DB::WNEstablishDisaggTaskHandler::execute(disaggregated::EstablishDisaggTaskResponse*)::$_4>, void (std::__1::unique_ptr<DB::DM::Remote::DisaggPhysicalTableReadSnapshot, std::__1::default
_delete<DB::DM::Remote::DisaggPhysicalTableReadSnapshot> > const&)>::operator()(std::__1::unique_ptr<DB::DM::Remote::DisaggPhysicalTableReadSnapshot, std::__1::default_delete<DB::DM::Remote::DisaggPhysicalTableReadSnapshot> > const&) [tiflash+137077922]\n                \t/data3/jaysonhuang/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345
\n       0x73401d3\tDB::DM::Remote::DisaggReadSnapshot::iterateTableSnapshots(std::__1::function<void (std::__1::unique_ptr<DB::DM::Remote::DisaggPhysicalTableReadSnapshot, std::__1::default_delete<DB::DM::Remote::DisaggPhysicalTableReadSnapshot> > const&)>) const [tiflash+120848851]\n                \tdbms/src/Storages/DeltaMerge/Remote/DisaggSnapshot.cpp:58\n
      0x82ba19c\tDB::WNEstablishDisaggTaskHandler::execute(disaggregated::EstablishDisaggTaskResponse*) [tiflash+137077148]\n                \tdbms/src/Flash/Disaggregated/WNEstablishDisaggTaskHandler.cpp:103\n       0x8000adb\tDB::FlashService::EstablishDisaggTask(grpc::ServerContext*, disaggregated::EstablishDisaggTaskRequest const*, disaggregated::EstablishDi
saggTaskResponse*) [tiflash+134220507]\n                \tdbms/src/Flash/FlashService.cpp:663\n       0x90f051e\tstd::__1::__function::__func<tikvpb::Tikv::Service::Service()::$_60, std::__1::allocator<tikvpb::Tikv::Service::Service()::$_60>, grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, disaggregated::EstablishDisaggTaskRequest const*, disaggregat
ed::EstablishDisaggTaskResponse*)>::operator()(tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, disaggregated::EstablishDisaggTaskRequest const*&&, disaggregated::EstablishDisaggTaskResponse*&&) [tiflash+151979294]\n                \t/data3/jaysonhuang/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345\n       0x9138bb5\tgrpc::Status grpc:
:internal::CatchingFunctionHandler<grpc::internal::RpcMethodHandler<tikvpb::Tikv::Service, disaggregated::EstablishDisaggTaskRequest, disaggregated::EstablishDisaggTaskResponse, google::protobuf::MessageLite, google::protobuf::MessageLite>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()>(grpc::internal::RpcMethodHandler<tikvpb::Ti
kv::Service, disaggregated::EstablishDisaggTaskRequest, disaggregated::EstablishDisaggTaskResponse, google::protobuf::MessageLite, google::protobuf::MessageLite>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()&&) [tiflash+152275893]\n                \tcontrib/grpc/include/grpcpp/impl/codegen/method_handler.h:44\n       0x91389d9\t
grpc::internal::RpcMethodHandler<tikvpb::Tikv::Service, disaggregated::EstablishDisaggTaskRequest, disaggregated::EstablishDisaggTaskResponse, google::protobuf::MessageLite, google::protobuf::MessageLite>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&) [tiflash+152275417]\n                \tcontrib/grpc/include/grpcpp/impl/codegen/method_hand
ler.h:113\n       0x8b2b929\tgrpc::Server::SyncRequest::ContinueRunAfterInterception() [tiflash+145930537]\n                \tcontrib/grpc/src/cpp/server/server_cc.cc:433\n       0x8b2b738\tgrpc::Server::SyncRequest::Run(std::__1::shared_ptr<grpc::Server::GlobalCallbacks> const&, bool) [tiflash+145930040]\n                \tcontrib/grpc/src/cpp/server/server_cc.cc:421\n       0x8b3b2d7\tgrpc::ThreadManager::MainWorkLoop() [tiflash+145994455]\n                \tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:211\n       0x8b3bba1\tgrpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::__invoke(void*) [tiflash+145996705]\n                \tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:36\n       0x8ecfe4c\tgrpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) [tiflash+149749324]\n                \tcontrib/grpc/src/core/lib/gprpp/thd_posix.cc:110\n  0x7fc92aa47609\tstart_thread [libpthread.so.0+34313]\n                \t/build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477\n  0x7fc92a88d133\tclone [libc.so.6+1175859]\n                \t/build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95"] [source="DisTaskId<MPP<query:<query_ts:1680788385645945105, local_query_id:754, server_id:3394589, start_ts:440608590553153598>,task_id:4>,executor=TableFullScan_59>"] [thread_id=192]