TiFlash crash for Address not mapped to object #4204
Assignees
Labels
Comments
|
We can see many error log like: 102578:[2022/03/08 20:02:21.171 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel28+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
102625:[2022/03/08 20:07:38.963 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
102879:[2022/03/08 20:08:42.387 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431683357167058946,task:3> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
102895:[2022/03/08 20:08:42.422 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
103048:[2022/03/08 20:09:14.660 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
103264:[2022/03/08 20:09:46.464 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
103402:[2022/03/08 20:10:18.701 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
105004:[2022/03/08 20:13:30.595 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
105165:[2022/03/08 20:14:02.785 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431683441026662453,task:3> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
105178:[2022/03/08 20:14:02.876 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
105368:[2022/03/08 20:15:07.558 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
105539:[2022/03/08 20:15:39.262 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
105595:[2022/03/08 20:16:11.469 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
106122:[2022/03/08 20:16:43.715 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
106783:[2022/03/08 20:17:15.523 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
107024:[2022/03/08 20:18:19.258 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
107553:[2022/03/08 20:18:51.282 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
108409:[2022/03/08 20:20:26.754 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
108989:[2022/03/08 20:22:34.566 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
109213:[2022/03/08 20:23:05.731 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
110491:[2022/03/08 20:26:18.959 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
111058:[2022/03/08 20:27:55.181 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
111166:[2022/03/08 20:28:27.456 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
111305:[2022/03/08 20:28:59.459 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
111830:[2022/03/08 20:30:34.156 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
112126:[2022/03/08 20:31:05.594 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
112363:[2022/03/08 20:31:36.657 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
112476:[2022/03/08 20:32:08.449 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
112601:[2022/03/08 20:32:40.397 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
113055:[2022/03/08 20:33:44.653 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
113347:[2022/03/08 20:34:16.774 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
113425:[2022/03/08 20:34:48.869 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
113566:[2022/03/08 20:35:21.251 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
113710:[2022/03/08 20:36:23.961 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431683792653516808,task:3> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
113726:[2022/03/08 20:36:23.999 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
113897:[2022/03/08 20:36:56.556 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
114055:[2022/03/08 20:37:28.169 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431683809574912069,task:3> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
114071:[2022/03/08 20:37:28.200 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
114152:[2022/03/08 20:37:59.367 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
114611:[2022/03/08 20:38:31.388 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
114975:[2022/03/08 20:39:02.563 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
115203:[2022/03/08 20:39:35.012 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
115338:[2022/03/08 20:40:06.288 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
115735:[2022/03/08 20:41:10.810 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431683867954643004,task:1> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
115751:[2022/03/08 20:41:10.845 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
115812:[2022/03/08 20:41:42.125 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
116050:[2022/03/08 20:42:14.351 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
116462:[2022/03/08 20:42:47.151 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
116522:[2022/03/08 20:43:18.264 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
117972:[2022/03/08 20:44:22.384 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
118326:[2022/03/08 20:45:26.737 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
118463:[2022/03/08 20:45:59.404 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
119094:[2022/03/08 20:47:03.957 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
119700:[2022/03/08 20:47:36.237 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
120061:[2022/03/08 20:49:11.959 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
120602:[2022/03/08 20:50:15.838 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
120922:[2022/03/08 20:51:18.561 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
121712:[2022/03/08 20:52:54.365 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::TiFlashException: Table 73 schema version 724 newer than query schema version 721 to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
121864:[2022/03/08 20:53:26.058 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::TiFlashException: Table 73 schema version 731 newer than query schema version 729 to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
121900:[2022/03/08 20:53:26.616 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
122214:[2022/03/08 20:55:01.756 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
122334:[2022/03/08 20:55:33.648 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
122441:[2022/03/08 20:56:04.857 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
122763:[2022/03/08 20:57:08.987 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
122851:[2022/03/08 20:57:40.471 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
123050:[2022/03/08 20:58:43.882 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
123285:[2022/03/08 20:59:15.053 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
123728:[2022/03/08 21:00:49.466 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431684176891346987,task:2> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
123744:[2022/03/08 21:00:49.505 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
124232:[2022/03/08 21:02:26.280 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
124275:[2022/03/08 21:02:57.931 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
124407:[2022/03/08 21:03:29.631 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
125025:[2022/03/08 21:04:02.101 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
125658:[2022/03/08 21:05:38.371 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
125760:[2022/03/08 21:06:10.482 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
125998:[2022/03/08 21:08:19.459 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
126408:[2022/03/08 21:09:55.459 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
126548:[2022/03/08 21:10:26.456 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
127242:[2022/03/08 21:12:01.250 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
127604:[2022/03/08 21:13:05.289 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431684369606508612,task:4> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
127617:[2022/03/08 21:13:05.653 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
128010:[2022/03/08 21:14:40.714 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
128128:[2022/03/08 21:15:12.940 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
128436:[2022/03/08 21:15:45.537 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
128723:[2022/03/08 21:16:49.684 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
129068:[2022/03/08 21:18:25.123 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::TiFlashException: Table 73 schema version 1109 newer than query schema version 1107 to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
129875:[2022/03/08 21:19:28.333 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Exchange receiver meet error : DB::TiFlashException: Table 73 schema version 1125 newer than query schema version 1124 to tunnel: tunnel8+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
130196:[2022/03/08 21:20:31.361 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
131987:[2022/03/08 21:24:50.764 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
132407:[2022/03/08 21:26:27.349 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
132762:[2022/03/08 21:27:31.155 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
133081:[2022/03/08 21:28:34.968 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431684613046534165,task:4> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
133097:[2022/03/08 21:28:35.189 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
133212:[2022/03/08 21:29:07.786 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
133520:[2022/03/08 21:30:42.919 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
133772:[2022/03/08 21:31:14.768 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431684655369683026,task:2> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
133789:[2022/03/08 21:31:14.853 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
134354:[2022/03/08 21:32:49.198 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
134958:[2022/03/08 21:34:58.090 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
136148:[2022/03/08 21:37:03.781 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::TiFlashException: Table 73 schema version 1388 newer than query schema version 1387 to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
136848:[2022/03/08 21:37:04.889 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431684747054546961,task:3> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
136864:[2022/03/08 21:37:04.962 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
137090:[2022/03/08 21:38:08.392 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
137218:[2022/03/08 21:39:11.851 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
137607:[2022/03/08 21:39:43.681 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Exchange receiver meet error : DB::Exception: Exchange receiver meet error : DB::TiFlashException: Table 73 schema version 1425 newer than query schema version 1423 to tunnel: tunnel17+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
138433:[2022/03/08 21:40:15.940 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
138588:[2022/03/08 21:40:48.165 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
138872:[2022/03/08 21:41:19.945 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
139090:[2022/03/08 21:41:51.160 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
139453:[2022/03/08 21:42:23.175 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431684830573625470,task:2> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
139469:[2022/03/08 21:42:23.260 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
139743:[2022/03/08 21:42:54.980 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431684838935756815,task:1> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
139760:[2022/03/08 21:42:55.014 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
140012:[2022/03/08 21:43:27.402 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
140138:[2022/03/08 21:43:59.866 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
140360:[2022/03/08 21:46:07.778 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
140722:[2022/03/08 21:47:43.773 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
143272:[2022/03/08 21:49:18.901 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
143377:[2022/03/08 21:49:50.180 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::TiFlashException: Table 73 schema version 1579 newer than query schema version 1578 to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
143507:[2022/03/08 21:50:55.572 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
143694:[2022/03/08 21:53:02.758 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
144030:[2022/03/08 21:54:05.874 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
144154:[2022/03/08 21:54:37.274 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
144557:[2022/03/08 21:55:08.460 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
144826:[2022/03/08 21:56:11.355 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
145309:[2022/03/08 21:59:23.882 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
145785:[2022/03/08 22:00:26.466 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Exchange receiver meet error : DB::TiFlashException: Table 73 schema version 1740 newer than query schema version 1737 to tunnel: tunnel7+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
146290:[2022/03/08 22:01:31.877 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
146734:[2022/03/08 22:02:35.382 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
147121:[2022/03/08 22:03:07.159 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
147362:[2022/03/08 22:03:38.233 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
147623:[2022/03/08 22:04:41.068 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
147907:[2022/03/08 22:05:43.951 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
147966:[2022/03/08 22:06:15.691 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
148136:[2022/03/08 22:06:47.539 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
148314:[2022/03/08 22:07:18.269 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Exchange receiver meet error : DB::TiFlashException: Table 73 schema version 1841 newer than query schema version 1839 to tunnel: tunnel5+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
148564:[2022/03/08 22:07:19.755 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
148633:[2022/03/08 22:07:52.036 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
148748:[2022/03/08 22:08:55.926 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
150079:[2022/03/08 22:10:00.087 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685264880697376,task:2> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
150096:[2022/03/08 22:10:00.177 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
150396:[2022/03/08 22:11:03.474 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685281369292818,task:1> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
150412:[2022/03/08 22:11:03.581 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
150478:[2022/03/08 22:11:34.557 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
156084: to tunnel: tunnel18+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
160787:[2022/03/08 22:12:44.167 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
161466:[2022/03/08 22:13:27.828 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685319406387275,task:2> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
161482:[2022/03/08 22:13:27.863 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
161615:[2022/03/08 22:14:32.898 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685336472223745,task:3> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
161631:[2022/03/08 22:14:32.933 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
161940:[2022/03/08 22:15:05.502 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685345005010980,task:3> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
161957:[2022/03/08 22:15:05.536 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
162030:[2022/03/08 22:15:37.915 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
163223:[2022/03/08 22:19:00.667 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
163732:[2022/03/08 22:20:05.581 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
163889:[2022/03/08 22:21:11.757 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
164388:[2022/03/08 22:22:16.991 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::TiFlashException: Table 73 schema version 2061 newer than query schema version 2060 to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
164531:[2022/03/08 22:23:22.570 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
164598:[2022/03/08 22:23:54.998 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
164913:[2022/03/08 22:25:34.138 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Exchange receiver meet error : DB::TiFlashException: Table 73 schema version 2106 newer than query schema version 2105 to tunnel: tunnel7+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
165023:[2022/03/08 22:25:35.854 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685510221266951,task:4> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
165039:[2022/03/08 22:25:35.902 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
166156:[2022/03/08 22:27:47.167 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
166220:[2022/03/08 22:28:20.456 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
166336:[2022/03/08 22:29:24.255 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Query was cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
166447:[2022/03/08 22:29:25.262 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
166724:[2022/03/08 22:30:32.101 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
166862:[2022/03/08 22:32:10.666 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
166962:[2022/03/08 22:33:15.690 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685630755078240,task:2> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
166979:[2022/03/08 22:33:15.726 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
167151:[2022/03/08 22:33:47.858 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
167250:[2022/03/08 22:34:53.464 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
167393:[2022/03/08 22:35:25.989 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
167590:[2022/03/08 22:36:30.916 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
170454:[2022/03/08 22:37:02.797 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
173436:[2022/03/08 22:37:08.806 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed, to tunnel: tunnel10+13: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
174355:[2022/03/08 22:37:33.429 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685698374860819,task:4> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
174372:[2022/03/08 22:37:33.462 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
174650:[2022/03/08 22:38:36.897 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
174906:[2022/03/08 22:39:41.284 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685731720364124,task:2> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
174922:[2022/03/08 22:39:41.569 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
175249:[2022/03/08 22:40:44.404 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685748392198186,task:4> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
175266:[2022/03/08 22:40:44.491 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
176347:[2022/03/08 22:42:53.107 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
176413:[2022/03/08 22:43:25.583 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685790413881345,task:4> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
176430:[2022/03/08 22:43:25.978 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
176544:[2022/03/08 22:43:58.572 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
176847:[2022/03/08 22:44:29.569 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
177135:[2022/03/08 22:45:00.816 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
177302:[2022/03/08 22:46:04.673 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685832396242980,task:3> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
177318:[2022/03/08 22:46:04.707 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
178207:[2022/03/08 22:47:42.065 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
178299:[2022/03/08 22:48:14.459 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685866435641372,task:4> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
178316:[2022/03/08 22:48:14.492 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
178698:[2022/03/08 22:48:46.819 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Exchange receiver meet error : DB::Exception: Exchange receiver meet error : DB::TiFlashException: Table 73 schema version 2450 newer than query schema version 2449 to tunnel: tunnel17+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
178963:[2022/03/08 22:48:47.590 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
179704:[2022/03/08 22:52:29.344 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685933177241668,task:1> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
179720:[2022/03/08 22:52:29.379 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
179830:[2022/03/08 22:53:34.095 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
180040:[2022/03/08 22:54:38.498 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431685967085830173,task:3> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
180056:[2022/03/08 22:54:38.530 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
180103:[2022/03/08 22:55:10.389 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel4+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
180414:[2022/03/08 22:55:42.880 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
180702:[2022/03/08 22:56:13.957 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Exchange receiver meet error : DB::TiFlashException: Table 73 schema version 2561 newer than query schema version 2560 to tunnel: tunnel6+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
180850:[2022/03/08 22:56:45.562 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
181103:[2022/03/08 22:58:51.817 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
181321:[2022/03/08 23:00:27.161 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
181560:[2022/03/08 23:02:03.097 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
181891:[2022/03/08 23:04:11.079 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
185186:[2022/03/08 23:06:51.923 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: task not in running state, may be cancelled to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
185271:[2022/03/08 23:07:23.664 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
185335:[2022/03/08 23:07:55.370 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431686175935954982,task:3> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
185352:[2022/03/08 23:07:55.483 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
185471:[2022/03/08 23:08:25.073 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
185785:[2022/03/08 23:08:26.367 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel1+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
186109:[2022/03/09 02:33:04.105 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: Consumer exits unexpected, grpc writes failed. to tunnel: tunnel2+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
|
102869:[2022/03/08 20:08:42.330 +08:00] [WARN] [<unknown>] ["TaskManager:Begin cancel query: 431683357167058946"] [thread_id=512]
102870:[2022/03/08 20:08:42.330 +08:00] [WARN] [<unknown>] ["TaskManager:Remaining task in query 431683357167058946 are: <query:431683357167058946,task:3> "] [thread_id=512]
102871:[2022/03/08 20:08:42.330 +08:00] [WARN] [MPPTask.cpp:396] ["MPPTask:<query:431683357167058946,task:3> Begin cancel task: <query:431683357167058946,task:3>"] [thread_id=27115]
102872:[2022/03/08 20:08:42.330 +08:00] [ERROR] [<unknown>] ["MPPTask:<query:431683357167058946,task:3> tunnel3+-1 grpc writes failed."] [thread_id=450]
102873:[2022/03/08 20:08:42.331 +08:00] [WARN] [<unknown>] ["MPPTask:<query:431683357167058946,task:3> Finish cancel task from running"] [thread_id=27115]
102874:[2022/03/08 20:08:42.331 +08:00] [WARN] [<unknown>] ["TaskManager:Finish cancel query: 431683357167058946"] [thread_id=512]
102875-[2022/03/08 20:08:42.335 +08:00] [WARN] [CoprocessorHandler.cpp:112] ["CoprocessorHandler:grpc::Status DB::CoprocessorHandler::execute(): LockException: region 705, message: "] [thread_id=344]
102876-[2022/03/08 20:08:42.335 +08:00] [WARN] [CoprocessorHandler.cpp:112] ["CoprocessorHandler:grpc::Status DB::CoprocessorHandler::execute(): LockException: region 2153, message: "] [thread_id=308]
102877-[2022/03/08 20:08:42.335 +08:00] [WARN] [CoprocessorHandler.cpp:112] ["CoprocessorHandler:grpc::Status DB::CoprocessorHandler::execute(): LockException: region 698, message: "] [thread_id=294]
102878-[2022/03/08 20:08:42.335 +08:00] [WARN] [CoprocessorHandler.cpp:112] ["CoprocessorHandler:grpc::Status DB::CoprocessorHandler::execute(): LockException: region 565, message: "] [thread_id=337]
102879:[2022/03/08 20:08:42.387 +08:00] [ERROR] [MPPTask.cpp:332] ["MPPTask:<query:431683357167058946,task:3> task running meets error: DB::Exception: write to tunnel which is already closed,grpc writes failed. Stack Trace : 0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
102880-1. /tiflash/tiflash(DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int)+0x26) [0x4d31a46]
102881-2. /tiflash/tiflash(DB::MPPTunnelBase<grpc_impl::ServerWriter<mpp::MPPDataPacket> >::write(mpp::MPPDataPacket const&, bool)+0x56f) [0xa8912bf]
102882-3. /tiflash/tiflash(DB::MPPTunnelSetBase<DB::MPPTunnel>::write(tipb::SelectResponse&)+0x34) [0xa893a54]
102883-4. /tiflash/tiflash(void DB::StreamingDAGResponseWriter<std::__1::shared_ptr<DB::MPPTunnelSet> >::batchWrite<true>()+0x77) [0xa854ff7]
102884-5. /tiflash/tiflash(DB::UnionBlockInputStream<(DB::StreamUnionMode)0, true>::readSuffix()+0x4a) [0xa84eeda]
102885-6. /tiflash/tiflash(DB::MPPTask::runImpl()+0x986) [0xa87ee16]
102886-7. /tiflash/tiflash(std::__1::__packaged_task_func<auto DB::wrapInvocable<std::__1::function<void ()>&>(bool, std::__1::function<void ()>&)::'lambda'(), std::__1::allocator<auto DB::wrapInvocable<std::__1::function<void ()>&>(bool, std::__1::function<void ()>&)::'lambda'()>, void ()>::operator()()+0x6e) [0x4dfe16e]
102887-8. /tiflash/tiflash(std::__1::packaged_task<void ()>::operator()()+0x45) [0x4dfe2d5]
102888-9. /tiflash/tiflash(DB::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > const&)+0x78) [0x4d92128]
102889-10. /tiflash/tiflash(DB::DynamicThreadPool::fixedWork(unsigned long)+0xb5) [0x4d91ae5]
102890-11. /tiflash/tiflash(auto std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...)::operator()<DB::DynamicThreadPool*, unsigned long>(auto&&...) const+0x121) [0x4d93151]
102891-12. /tiflash/tiflash() [0x4d92f79]
102892-13. /lib64/libpthread.so.0(+0x7ea5) [0x7f3ab86c8ea5]
102893-14. /lib64/libc.so.6(clone+0x6d) [0x7f3ab81db96d]
102894-"] [thread_id=135]
102895-[2022/03/08 20:08:42.422 +08:00] [ERROR] [Exception.cpp:64] ["MPPTask:Failed to write error DB::Exception: write to tunnel which is already closed,grpc writes failed. to tunnel: tunnel3+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:
102896-
102897-0. /tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x4d34443]
102898-1. /tiflash/tiflash(DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int)+0x26) [0x4d31a46]
102899-2. /tiflash/tiflash(DB::MPPTunnelBase<grpc_impl::ServerWriter<mpp::MPPDataPacket> >::write(mpp::MPPDataPacket const&, bool)+0x56f) [0xa8912bf]
102900-3. /tiflash/tiflash(DB::MPPTask::writeErrToAllTunnels(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)+0x76) [0xa8801e6]
102901-4. /tiflash/tiflash(DB::MPPTask::runImpl()+0xb49) [0xa87efd9]
102902-5. /tiflash/tiflash(std::__1::__packaged_task_func<auto DB::wrapInvocable<std::__1::function<void ()>&>(bool, std::__1::function<void ()>&)::'lambda'(), std::__1::allocator<auto DB::wrapInvocable<std::__1::function<void ()>&>(bool, std::__1::function<void ()>&)::'lambda'()>, void ()>::operator()()+0x6e) [0x4dfe16e]
102903-6. /tiflash/tiflash(std::__1::packaged_task<void ()>::operator()()+0x45) [0x4dfe2d5]
102904-7. /tiflash/tiflash(DB::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > const&)+0x78) [0x4d92128]
102905-8. /tiflash/tiflash(DB::DynamicThreadPool::fixedWork(unsigned long)+0xb5) [0x4d91ae5]
102906-9. /tiflash/tiflash(auto std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...)::operator()<DB::DynamicThreadPool*, unsigned long>(auto&&...) const+0x121) [0x4d93151]
102907-10. /tiflash/tiflash() [0x4d92f79]
102908-11. /lib64/libpthread.so.0(+0x7ea5) [0x7f3ab86c8ea5]
102909-12. /lib64/libc.so.6(clone+0x6d) [0x7f3ab81db96d]
102910-"] [thread_id=135]
102911:[2022/03/08 20:08:42.422 +08:00] [ERROR] [<unknown>] ["TaskManager:The task <query:431683357167058946,task:3> cannot be found and fail to unregister"] [thread_id=135]
102912:[2022/03/08 20:08:42.422 +08:00] [WARN] [<unknown>] ["MPPTask:<query:431683357167058946,task:3> finish task which was cancelled before"] [thread_id=135] |
|
The error is expected, since the query is cancelled before: |
12 tasks
ti-chi-bot
pushed a commit
that referenced
this issue
Mar 16, 2022
JaySon-Huang
pushed a commit
to JaySon-Huang/tiflash
that referenced
this issue
Mar 17, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
2. What did you expect to see? (Required)
3. What did you see instead (Required)
4. What is your TiFlash version? (Required)
master
4d8a1be
The text was updated successfully, but these errors were encountered: