upgrade from v4.0.8 to v5.1.0 fail, report tiflash error

[seiya-annie]

1. error from tiup

[ Serial ] - InitConfig: cluster=local, user=tidb, host=172.16.5.101, path=/home/tidb/.tiup/storage/cluster/clusters/local/config-cache/tidb-4000.service, deploy_dir=/home/tidb/deploy/tidb-4000, data_dir=[], log_dir=/home/tidb/deploy/tidb-4000/log, cache_dir=/home/tidb/.tiup/storage/cluster/clusters/local/config-cache

Error: stderr: Run Command Timeout!
: executor.ssh.execute_timedout: Execute command over SSH timedout for '[email protected]:22' {ssh_stderr: Run Command Timeout!
, ssh_stdout: , ssh_command: export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin tar --no-same-owner -zxf /home/tidb/deploy/tiflash-9000/bin/tiflash-v5.1.0-linux-amd64.tar.gz -C /home/tidb/deploy/tiflash-9000/bin && rm /home/tidb/deploy/tiflash-9000/bin/tiflash-v5.1.0-linux-amd64.tar.gz}

Verbose debug logs has been written to /home/tidb/.tiup/logs/tiup-cluster-debug-2021-06-22-13-42-20.log.
Error: run `/home/tidb/.tiup/components/cluster/v1.5.1/tiup-cluster` (wd:/home/tidb/.tiup/data/Sb2V5xM) failed: exit status 1

2. errors found in tiflash.log

[2021/06/22 13:36:56.482 +08:00] [ERROR] [<unknown>] ["DAGDriver: void DB::DAGDriver<batch>::execute() [with bool batch = true]: [FLASH:Coprocessor:Internal] TiFlash server is terminating\n0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x3408075]\n1. bin/tiflash/tiflash(DB::TiFlashException::TiFlashException(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::TiFlashError const&)+0x34) [0x3cd20a4]\n2. bin/tiflash/tiflash(DB::DAGQueryBlockInterpreter::executeTS(tipb::TableScan const&, DB::Pipeline&)+0x2ba4) [0x7014514]\n3. bin/tiflash/tiflash(DB::DAGQueryBlockInterpreter::executeImpl(DB::Pipeline&)+0x81) [0x7015c31]\n4. bin/tiflash/tiflash(DB::DAGQueryBlockInterpreter::execute()+0x35) [0x7016165]\n5. bin/tiflash/tiflash(DB::InterpreterDAG::executeQueryBlock(DB::DAGQueryBlock&, std::vector<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, DB::SubqueryForSet, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, DB::SubqueryForSet> > >, std::allocator<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, DB::SubqueryForSet, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, DB::SubqueryForSet> > > > >&)+0xdca) [0x6fe1e3a]\n6. bin/tiflash/tiflash(DB::InterpreterDAG::execute()+0x77) [0x6fe26d7]\n7. bin/tiflash/tiflash() [0x67b82d6]\n8. bin/tiflash/tiflash(DB::executeQuery(DB::DAGQuerySource&, DB::Context&, bool, DB::QueryProcessingStage::Enum)+0x63) [0x67b92d3]\n9. bin/tiflash/tiflash(DB::DAGDriver<true>::execute()+0x194) [0x6fc7094]\n10. bin/tiflash/tiflash(DB::BatchCoprocessorHandler::execute()+0x49a) [0x6ff46ba]\n11. bin/tiflash/tiflash(DB::FlashService::BatchCoprocessor(grpc_impl::ServerContext*, coprocessor::BatchRequest const*, grpc_impl::ServerWriter<coprocessor::BatchResponse>*)+0x689) [0x6faf649]\n12. bin/tiflash/tiflash(std::_Function_handler<grpc::Status (tikvpb::Tikv::Service*, grpc_impl::ServerContext*, coprocessor::BatchRequest const*, grpc_impl::ServerWriter<coprocessor::BatchResponse>*), std::_Mem_fn<grpc::Status (tikvpb::Tikv::Service::*)(grpc_impl::ServerContext*, coprocessor::BatchRequest const*, grpc_impl::ServerWriter<coprocessor::BatchResponse>*)> >::_M_invoke(std::_Any_data const&, tikvpb::Tikv::Service*&&, grpc_impl::ServerContext*&&, coprocessor::BatchRequest const*&&, grpc_impl::ServerWriter<coprocessor::BatchResponse>*&&)+0x38) [0x77507e8]\n13. bin/tiflash/tiflash(grpc::Status grpc::internal::CatchingFunctionHandler<grpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, coprocessor::BatchRequest, coprocessor::BatchResponse>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}>(grpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, coprocessor::BatchRequest, coprocessor::BatchResponse>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}&&)+0x53) [0x77a6033]\n14. bin/tiflash/tiflash(grpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, coprocessor::BatchRequest, coprocessor::BatchResponse>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)+0xac) [0x77a618c]\n15. bin/tiflash/tiflash(grpc_impl::Server::SyncRequest::CallData::ContinueRunAfterInterception()+0x160) [0x782d890]\n16. bin/tiflash/tiflash(grpc_impl::Server::SyncRequestThreadManager::DoWork(void*, bool, bool)+0x42f) [0x782ee4f]\n17. bin/tiflash/tiflash(grpc::ThreadManager::MainWorkLoop()+0x9a) [0x7834a8a]\n18. bin/tiflash/tiflash(grpc::ThreadManager::WorkerThread::Run()+0xb) [0x7834b6b]\n19. bin/tiflash/tiflash() [0x7babc32]\n20. /lib64/libpthread.so.0(+0x7ea4) [0x7eff1aae1ea4]\n21. /lib64/libc.so.6(clone+0x6c) [0x7eff1a50896c]\n"] [thread_id=5077]

3. errors found in tiflash_err.log

2021.06.22 13:36:54.837975 [ 5086 ] <Warning> pingcap.index_read: Region {103,6,27} get index failed, error message is: Exception: ReadIndex Failed14: failed to connect to all addresses
2021.06.22 13:36:55.388586 [ 5083 ] <Warning> pingcap.index_read: Region {130,6,27} get index failed, error message is: Exception: ReadIndex Failed14: failed to connect to all addresses
2021.06.22 13:36:55.961321 [ 5084 ] <Warning> pingcap.index_read: Region {139,6,27} get index failed, error message is: Exception: ReadIndex Failed14: failed to connect to all addresses
2021.06.22 13:36:56.435193 [ 5085 ] <Warning> pingcap.index_read: Region {121,6,27} get index failed, error message is: Exception: ReadIndex Failed14: failed to connect to all addresses
2021.06.22 13:36:56.482586 [ 5077 ] <Error> DAGDriver: void DB::DAGDriver<batch>::execute() [with bool batch = true]: [FLASH:Coprocessor:Internal] TiFlash server is terminating
0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x15) [0x3408075]
1. bin/tiflash/tiflash(DB::TiFlashException::TiFlashException(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::TiFlashError const&)+0x34) [0x3cd20a4]
2. bin/tiflash/tiflash(DB::DAGQueryBlockInterpreter::executeTS(tipb::TableScan const&, DB::Pipeline&)+0x2ba4) [0x7014514]
3. bin/tiflash/tiflash(DB::DAGQueryBlockInterpreter::executeImpl(DB::Pipeline&)+0x81) [0x7015c31]
4. bin/tiflash/tiflash(DB::DAGQueryBlockInterpreter::execute()+0x35) [0x7016165]
5. bin/tiflash/tiflash(DB::InterpreterDAG::executeQueryBlock(DB::DAGQueryBlock&, 
5. bin/tiflash/tiflash(DB::InterpreterDAG::executeQueryBlock(DB::DAGQueryBlock&, std::vector<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, DB::SubqueryForSet, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, DB::SubqueryForSet> > >, std::allocator<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, DB::SubqueryForSet, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, DB::SubqueryForSet> > > > >&)+0xdca) [0x6fe1e3a]
6. bin/tiflash/tiflash(DB::InterpreterDAG::execute()+0x77) [0x6fe26d7]
7. bin/tiflash/tiflash() [0x67b82d6]
8. bin/tiflash/tiflash(DB::executeQuery(DB::DAGQuerySource&, DB::Context&, bool, DB::QueryProcessingStage::Enum)+0x63) [0x67b92d3]
9. bin/tiflash/tiflash(DB::DAGDriver<true>::execute()+0x194) [0x6fc7094]
10. bin/tiflash/tiflash(DB::BatchCoprocessorHandler::execute()+0x49a) [0x6ff46ba]
11. bin/tiflash/tiflash(DB::FlashService::BatchCoprocessor(grpc_impl::ServerContext*, coprocessor::BatchRequest const*, grpc_impl::ServerWriter<coprocessor::BatchResponse>*)+0x689) [0x6faf649]
12. bin/tiflash/tiflash(std::_Function_handler<grpc::Status (tikvpb::Tikv::Service*, grpc_impl::ServerContext*, coprocessor::BatchRequest const*, grpc_impl::ServerWriter<coprocessor::BatchResponse>*), std::_Mem_fn<grpc::Status (tikvpb::Tikv::Service::*)(grpc_impl::ServerContext*, coprocessor::BatchRequest const*, grpc_impl::ServerWriter<coprocessor::BatchResponse>*)> >::_M_invoke(std::_Any_data const&, tikvpb::Tikv::Service*&&, grpc_impl::ServerContext*&&, coprocessor::BatchRequest const*&&, grpc_impl::ServerWriter<coprocessor::BatchResponse>*&&)+0x38) [0x77507e8]
13. bin/tiflash/tiflash(grpc::Status grpc::internal::CatchingFunctionHandler<grpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, coprocessor::BatchRequest, coprocessor::BatchResponse>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}>(grpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, coprocessor::BatchRequest, coprocessor::BatchResponse>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::{lambda()#1}&&)+0x53) [0x77a6033]
14. bin/tiflash/tiflash(grpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, coprocessor::BatchRequest, coprocessor::BatchResponse>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)+0xac) [0x77a618c]
15. bin/tiflash/tiflash(grpc_impl::Server::SyncRequest::CallData::ContinueRunAfterInterception()+0x160) [0x782d890]
16. bin/tiflash/tiflash(grpc_impl::Server::SyncRequestThreadManager::DoWork(void*, bool, bool)+0x42f) [0x782ee4f]
17. bin/tiflash/tiflash(grpc::ThreadManager::MainWorkLoop()+0x9a) [0x7834a8a]
18. bin/tiflash/tiflash(grpc::ThreadManager::WorkerThread::Run()+0xb) [0x7834b6b]
19. bin/tiflash/tiflash() [0x7babc32]
20. /lib64/libpthread.so.0(+0x7ea4) [0x7eff1aae1ea4]
21. /lib64/libc.so.6(clone+0x6c) [0x7eff1a50896c]

2021.06.22 13:48:48.396269 [ 1 ] <Warning> Application: The configuration "path" is deprecated. Check [storage] section for new style.

[seiya-annie]

triggered by tiup easyssh issue
appleboy/easyssh-proxy#68
pingcap/tiup#1433