Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](arrow-flight-sql) Fix get Schema failed when enable_parallel_result_sink is false #37779

Merged
merged 4 commits into from
Jul 16, 2024
Merged

[fix](arrow-flight-sql) Fix get Schema failed when enable_parallel_result_sink is false #37779

merged 4 commits into from
Jul 16, 2024

Conversation

xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Jul 15, 2024 •
edited
Loading

if enable_parallel_result_sink is true, use queryID to get Schema and fetch Result.
if enable_parallel_result_sink is false, use fragment 0 instance ID to get Schema and fetch Result.

Caused by: cfjd.org.apache.arrow.flight.FlightRuntimeException: not found arrow flight schema, maybe query has been canceled, queryid: cef21a3bd35e40c5-bd25643a1e02dbb7
	at cfjd.org.apache.arrow.flight.CallStatus.toRuntimeException(CallStatus.java:131)
	at cfjd.org.apache.arrow.flight.grpc.StatusUtils.fromGrpcRuntimeException(StatusUtils.java:164)
	at cfjd.org.apache.arrow.flight.grpc.StatusUtils.fromThrowable(StatusUtils.java:185)
	at cfjd.org.apache.arrow.flight.FlightStream$Observer.onError(FlightStream.java:456)
	at cfjd.org.apache.arrow.flight.FlightClient$1.onError(FlightClient.java:350)
	at cfjd.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481)
	at cfjd.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at cfjd.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at cfjd.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at cfjd.org.apache.arrow.flight.grpc.ClientInterceptorAdapter$FlightClientCallListener.onClose(ClientInterceptorAdapter.java:117)
	at cfjd.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574)
	at cfjd.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72)
	at cfjd.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
	at cfjd.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
	at cfjd.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at cfjd.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40115 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a91b7f1ee9820c9e4080836dfd90afd6e50c7a69, data reload: false

------ Round 1 ----------------------------------
q1	18373	4595	4389	4389
q2	2592	201	198	198
q3	10718	1187	1071	1071
q4	10538	721	765	721
q5	7578	2750	2736	2736
q6	229	145	146	145
q7	988	633	615	615
q8	9341	2118	2104	2104
q9	8839	6584	6562	6562
q10	8847	3725	3736	3725
q11	443	239	243	239
q12	395	241	229	229
q13	17893	3010	3046	3010
q14	280	247	237	237
q15	526	478	490	478
q16	518	411	380	380
q17	962	703	671	671
q18	8001	7526	7426	7426
q19	8730	1517	1353	1353
q20	673	318	320	318
q21	5447	3219	3274	3219
q22	356	295	289	289
Total cold run time: 122267 ms
Total hot run time: 40115 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4432	4282	4275	4275
q2	371	287	263	263
q3	3022	2817	2734	2734
q4	1885	1582	1600	1582
q5	5285	5330	5297	5297
q6	223	136	132	132
q7	2125	1758	1698	1698
q8	3198	3355	3270	3270
q9	8435	8441	8356	8356
q10	3918	3670	3743	3670
q11	602	490	490	490
q12	767	576	574	574
q13	16283	2985	2978	2978
q14	302	290	263	263
q15	518	477	480	477
q16	475	411	440	411
q17	1771	1502	1457	1457
q18	7526	7372	7306	7306
q19	1722	1572	1455	1455
q20	1985	1771	1754	1754
q21	4868	4709	4817	4709
q22	587	512	468	468
Total cold run time: 70300 ms
Total hot run time: 53619 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172184 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a91b7f1ee9820c9e4080836dfd90afd6e50c7a69, data reload: false

query1	908	381	362	362
query2	6455	1877	1789	1789
query3	6650	208	212	208
query4	26058	17459	17433	17433
query5	4170	481	476	476
query6	271	162	176	162
query7	4597	280	286	280
query8	235	193	199	193
query9	8459	2377	2358	2358
query10	426	280	266	266
query11	11194	10003	10157	10003
query12	139	85	82	82
query13	1641	361	359	359
query14	10124	7627	7907	7627
query15	216	165	165	165
query16	7798	307	316	307
query17	1816	549	518	518
query18	1970	275	270	270
query19	208	148	150	148
query20	90	82	79	79
query21	207	130	123	123
query22	4292	4185	3997	3997
query23	33708	33001	33203	33001
query24	11901	2876	2821	2821
query25	646	359	370	359
query26	1727	149	146	146
query27	2835	269	270	269
query28	7164	1973	1972	1972
query29	1068	649	599	599
query30	288	149	148	148
query31	957	746	745	745
query32	101	53	56	53
query33	769	301	295	295
query34	945	482	481	481
query35	675	580	564	564
query36	1132	944	940	940
query37	158	80	78	78
query38	2871	2809	2735	2735
query39	875	802	825	802
query40	286	124	121	121
query41	47	46	45	45
query42	119	101	105	101
query43	499	480	453	453
query44	1204	748	735	735
query45	199	164	163	163
query46	1091	747	741	741
query47	1864	1772	1761	1761
query48	368	334	293	293
query49	1158	411	417	411
query50	786	405	393	393
query51	6918	6745	6835	6745
query52	110	93	93	93
query53	363	294	286	286
query54	1048	446	440	440
query55	75	74	72	72
query56	287	264	280	264
query57	1178	1075	1058	1058
query58	268	244	257	244
query59	2891	2606	2532	2532
query60	308	286	313	286
query61	97	96	91	91
query62	824	668	666	666
query63	323	289	298	289
query64	10502	2201	1657	1657
query65	3176	3107	3114	3107
query66	1399	356	339	339
query67	15389	14868	15150	14868
query68	4566	533	526	526
query69	469	338	319	319
query70	1199	1135	1188	1135
query71	367	277	292	277
query72	7008	5242	6393	5242
query73	738	324	327	324
query74	6220	5730	5672	5672
query75	3345	2710	2729	2710
query76	2581	937	959	937
query77	468	309	305	305
query78	9537	9493	8953	8953
query79	2363	514	515	514
query80	1881	461	468	461
query81	560	217	222	217
query82	853	136	134	134
query83	275	164	164	164
query84	275	91	90	90
query85	2122	315	337	315
query86	497	299	324	299
query87	3277	3127	3146	3127
query88	4282	2473	2470	2470
query89	476	388	379	379
query90	1840	195	195	195
query91	140	113	114	113
query92	69	50	53	50
query93	2625	504	494	494
query94	1339	223	215	215
query95	407	315	314	314
query96	602	283	270	270
query97	3148	3072	3046	3046
query98	245	199	196	196
query99	1790	1323	1254	1254
Total cold run time: 284085 ms
Total hot run time: 172184 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.45 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a91b7f1ee9820c9e4080836dfd90afd6e50c7a69, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.03
query3	0.22	0.05	0.05
query4	1.67	0.08	0.08
query5	0.50	0.48	0.48
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.53	0.54	0.55
query11	0.15	0.11	0.12
query12	0.15	0.13	0.12
query13	0.59	0.58	0.58
query14	0.74	0.78	0.79
query15	0.86	0.81	0.80
query16	0.35	0.37	0.38
query17	0.98	1.01	1.04
query18	0.22	0.22	0.22
query19	1.92	1.75	1.81
query20	0.02	0.01	0.01
query21	15.41	0.74	0.66
query22	5.14	6.84	1.58
query23	18.27	1.45	1.40
query24	2.12	0.23	0.22
query25	0.16	0.08	0.08
query26	0.30	0.21	0.20
query27	0.46	0.23	0.23
query28	13.23	1.03	1.00
query29	12.61	3.34	3.37
query30	0.25	0.06	0.05
query31	2.86	0.39	0.39
query32	3.27	0.47	0.46
query33	2.92	2.86	2.92
query34	16.97	4.32	4.35
query35	4.47	4.38	4.43
query36	0.66	0.46	0.51
query37	0.19	0.16	0.16
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.15	0.13	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.6 s
Total hot run time: 30.45 s

@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39646 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f8750575b4c5e2129a47c8fdb3fb34d903d551f, data reload: false

------ Round 1 ----------------------------------
q1	17621	4354	4271	4271
q2	2022	189	183	183
q3	10454	1237	1122	1122
q4	10210	790	820	790
q5	7533	2682	2635	2635
q6	218	137	138	137
q7	949	591	589	589
q8	9224	2073	2055	2055
q9	8540	6542	6546	6542
q10	8612	3794	3756	3756
q11	461	230	241	230
q12	389	221	219	219
q13	17793	2982	2932	2932
q14	282	235	237	235
q15	537	487	495	487
q16	477	395	375	375
q17	960	627	693	627
q18	8426	7445	7357	7357
q19	4576	1431	1370	1370
q20	693	346	321	321
q21	4924	3171	3128	3128
q22	342	289	285	285
Total cold run time: 115243 ms
Total hot run time: 39646 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4320	4258	4240	4240
q2	367	255	263	255
q3	3006	2785	2727	2727
q4	1874	1648	1617	1617
q5	5241	5303	5287	5287
q6	216	132	134	132
q7	2068	1741	1724	1724
q8	3192	3326	3285	3285
q9	8454	8400	8380	8380
q10	3917	3708	3693	3693
q11	594	476	485	476
q12	804	607	584	584
q13	17544	2970	2990	2970
q14	299	271	268	268
q15	513	476	482	476
q16	479	422	432	422
q17	1781	1459	1480	1459
q18	7568	7601	7349	7349
q19	2860	1404	1636	1404
q20	1978	1785	1774	1774
q21	4887	4711	4624	4624
q22	561	503	501	501
Total cold run time: 72523 ms
Total hot run time: 53647 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172209 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f8750575b4c5e2129a47c8fdb3fb34d903d551f, data reload: false

query1	905	366	368	366
query2	6459	1983	1898	1898
query3	6649	203	212	203
query4	27573	17413	17361	17361
query5	4216	486	481	481
query6	277	160	156	156
query7	4580	285	283	283
query8	236	201	186	186
query9	8521	2359	2337	2337
query10	434	304	267	267
query11	10473	9985	9983	9983
query12	132	86	83	83
query13	1661	364	360	360
query14	10055	7648	6863	6863
query15	210	171	166	166
query16	7815	303	304	303
query17	1783	539	535	535
query18	1950	269	273	269
query19	190	149	175	149
query20	91	81	81	81
query21	206	135	122	122
query22	4456	4171	4020	4020
query23	33687	33669	33048	33048
query24	12135	2949	2884	2884
query25	598	372	388	372
query26	1558	148	150	148
query27	2981	270	273	270
query28	7671	2004	1953	1953
query29	916	611	590	590
query30	288	153	145	145
query31	976	739	727	727
query32	91	52	53	52
query33	767	287	298	287
query34	1018	472	484	472
query35	685	575	578	575
query36	1068	910	955	910
query37	241	81	77	77
query38	2879	2744	2784	2744
query39	882	780	823	780
query40	274	142	118	118
query41	49	48	43	43
query42	128	98	101	98
query43	509	474	490	474
query44	1235	737	726	726
query45	193	160	162	160
query46	1077	713	730	713
query47	1860	1780	1777	1777
query48	369	302	298	298
query49	1197	408	411	408
query50	773	393	394	393
query51	6897	6921	6822	6822
query52	110	91	102	91
query53	361	284	291	284
query54	995	447	437	437
query55	74	71	71	71
query56	282	276	276	276
query57	1167	1059	1031	1031
query58	265	255	279	255
query59	3108	2789	2843	2789
query60	306	291	268	268
query61	94	96	93	93
query62	811	652	623	623
query63	315	282	284	282
query64	10431	2202	7423	2202
query65	3170	3126	3095	3095
query66	1352	341	325	325
query67	15357	15108	14841	14841
query68	4609	539	529	529
query69	473	314	335	314
query70	1190	1112	1085	1085
query71	412	285	275	275
query72	6929	5596	5924	5596
query73	739	324	315	315
query74	6103	5718	5644	5644
query75	3442	2719	2676	2676
query76	2765	894	893	893
query77	488	305	303	303
query78	9550	9053	8935	8935
query79	2160	509	505	505
query80	1649	472	465	465
query81	584	214	221	214
query82	819	130	140	130
query83	284	167	161	161
query84	266	86	89	86
query85	1235	375	297	297
query86	435	319	336	319
query87	3280	3114	3155	3114
query88	3883	2426	2420	2420
query89	473	371	372	371
query90	1876	189	195	189
query91	126	98	97	97
query92	67	51	52	51
query93	2143	501	488	488
query94	1222	210	211	210
query95	396	313	312	312
query96	589	269	270	269
query97	3227	3023	3044	3023
query98	229	198	188	188
query99	1607	1258	1236	1236
Total cold run time: 283322 ms
Total hot run time: 172209 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f8750575b4c5e2129a47c8fdb3fb34d903d551f, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.69	0.07	0.08
query5	0.50	0.49	0.47
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.55	0.54	0.55
query11	0.14	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.76	0.78	0.79
query15	0.85	0.81	0.82
query16	0.37	0.37	0.36
query17	1.04	0.97	1.05
query18	0.22	0.21	0.22
query19	1.74	1.70	1.69
query20	0.03	0.01	0.01
query21	15.40	0.73	0.65
query22	4.32	6.45	2.67
query23	18.31	1.33	1.29
query24	2.13	0.22	0.22
query25	0.15	0.10	0.09
query26	0.29	0.21	0.20
query27	0.46	0.22	0.22
query28	13.32	1.03	1.00
query29	12.58	3.31	3.24
query30	0.27	0.07	0.05
query31	2.87	0.38	0.39
query32	3.28	0.48	0.47
query33	2.90	2.91	2.86
query34	16.90	4.33	4.36
query35	4.45	4.42	4.41
query36	0.64	0.47	0.48
query37	0.18	0.16	0.15
query38	0.17	0.15	0.14
query39	0.04	0.04	0.04
query40	0.15	0.12	0.13
query41	0.10	0.05	0.06
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.72 s
Total hot run time: 31.23 s

@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 15, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40294 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 637b2474fee7e26befdff7cf20d6311a328b4b17, data reload: false

------ Round 1 ----------------------------------
q1	17605	4433	4312	4312
q2	2013	192	184	184
q3	10524	1280	1088	1088
q4	10985	866	836	836
q5	7987	2767	2662	2662
q6	222	137	134	134
q7	994	609	602	602
q8	9592	2157	2168	2157
q9	9099	6701	6684	6684
q10	8911	3819	3810	3810
q11	455	234	239	234
q12	409	228	217	217
q13	19070	2957	2981	2957
q14	274	230	245	230
q15	529	473	482	473
q16	512	394	381	381
q17	985	727	668	668
q18	8277	7495	7500	7495
q19	4150	1449	1401	1401
q20	682	311	321	311
q21	4918	3170	3226	3170
q22	352	298	288	288
Total cold run time: 118545 ms
Total hot run time: 40294 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4391	4305	4288	4288
q2	366	279	267	267
q3	3029	2785	2722	2722
q4	1902	1609	1643	1609
q5	5273	5288	5305	5288
q6	222	128	127	127
q7	2102	1701	1700	1700
q8	3168	3349	3305	3305
q9	8466	8425	8420	8420
q10	3863	3708	3671	3671
q11	577	479	488	479
q12	778	602	592	592
q13	16341	2978	2979	2978
q14	311	266	260	260
q15	510	470	467	467
q16	459	401	415	401
q17	1791	1491	1475	1475
q18	7694	7469	7355	7355
q19	1714	1540	1543	1540
q20	1979	1764	1775	1764
q21	4853	4771	4771	4771
q22	598	493	514	493
Total cold run time: 70387 ms
Total hot run time: 53972 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172138 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 637b2474fee7e26befdff7cf20d6311a328b4b17, data reload: false

query1	908	371	379	371
query2	6463	1921	1893	1893
query3	6649	208	217	208
query4	28570	17467	17506	17467
query5	4162	478	485	478
query6	266	192	165	165
query7	4596	287	297	287
query8	244	206	199	199
query9	8532	2442	2426	2426
query10	451	287	270	270
query11	10771	10008	10236	10008
query12	128	84	83	83
query13	1627	362	368	362
query14	10198	7539	7514	7514
query15	223	167	167	167
query16	7809	310	302	302
query17	1826	553	522	522
query18	1967	270	269	269
query19	192	147	144	144
query20	92	80	80	80
query21	206	127	126	126
query22	4231	4058	4115	4058
query23	33548	32963	33167	32963
query24	11932	2864	2889	2864
query25	665	362	388	362
query26	1774	153	147	147
query27	2987	272	269	269
query28	7444	2054	2050	2050
query29	1082	607	605	605
query30	288	153	147	147
query31	971	743	739	739
query32	90	52	53	52
query33	782	300	284	284
query34	948	479	476	476
query35	674	563	561	561
query36	1087	950	954	950
query37	193	85	81	81
query38	2902	2750	2830	2750
query39	859	797	841	797
query40	274	118	120	118
query41	50	45	46	45
query42	116	102	99	99
query43	533	486	466	466
query44	1184	748	722	722
query45	196	161	163	161
query46	1096	739	734	734
query47	1901	1767	1769	1767
query48	375	288	324	288
query49	1194	428	407	407
query50	788	396	405	396
query51	6930	6794	6863	6794
query52	103	91	95	91
query53	361	294	297	294
query54	967	451	445	445
query55	74	74	73	73
query56	295	268	269	268
query57	1121	1078	1042	1042
query58	254	246	247	246
query59	2894	2668	2691	2668
query60	297	273	270	270
query61	97	97	93	93
query62	870	649	663	649
query63	324	287	287	287
query64	10478	2215	1672	1672
query65	3165	3103	3089	3089
query66	1299	332	331	331
query67	15426	15165	14741	14741
query68	4618	558	538	538
query69	480	326	320	320
query70	1106	1120	1120	1120
query71	409	277	277	277
query72	7345	5581	5180	5180
query73	724	325	327	325
query74	6081	5665	5682	5665
query75	3423	2702	2677	2677
query76	2815	912	914	912
query77	500	304	299	299
query78	10226	8946	8904	8904
query79	3195	535	516	516
query80	1437	470	468	468
query81	560	218	224	218
query82	900	137	138	137
query83	194	168	168	168
query84	282	88	89	88
query85	1495	342	300	300
query86	465	313	304	304
query87	3305	3110	3148	3110
query88	4097	2389	2385	2385
query89	497	382	367	367
query90	1885	187	186	186
query91	125	101	102	101
query92	137	49	50	49
query93	4355	522	516	516
query94	1216	207	209	207
query95	406	313	318	313
query96	604	277	267	267
query97	3187	3018	3001	3001
query98	225	200	200	200
query99	1573	1311	1268	1268
Total cold run time: 288866 ms
Total hot run time: 172138 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.5 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 637b2474fee7e26befdff7cf20d6311a328b4b17, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.08	0.09
query5	0.51	0.50	0.48
query6	1.12	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.57	0.49	0.50
query10	0.54	0.54	0.54
query11	0.15	0.11	0.11
query12	0.14	0.13	0.12
query13	0.59	0.58	0.58
query14	0.76	0.77	0.77
query15	0.85	0.83	0.82
query16	0.35	0.37	0.37
query17	0.99	0.99	0.94
query18	0.21	0.22	0.23
query19	1.78	1.69	1.67
query20	0.01	0.01	0.01
query21	15.50	0.76	0.67
query22	4.79	6.64	1.87
query23	18.30	1.43	1.25
query24	2.10	0.23	0.23
query25	0.16	0.08	0.08
query26	0.30	0.21	0.21
query27	0.45	0.23	0.23
query28	13.33	1.02	0.99
query29	12.63	3.33	3.29
query30	0.26	0.06	0.06
query31	2.87	0.38	0.39
query32	3.26	0.46	0.47
query33	2.89	2.92	2.92
query34	17.17	4.35	4.36
query35	4.39	4.39	4.50
query36	0.65	0.46	0.47
query37	0.19	0.15	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 110.42 s
Total hot run time: 30.5 s

@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39674 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5de8d0bb20c50032fe6158f1b26ae8463b498d4f, data reload: false

------ Round 1 ----------------------------------
q1	17610	4316	4230	4230
q2	2017	189	184	184
q3	10451	1152	1078	1078
q4	10177	788	749	749
q5	7535	2656	2648	2648
q6	217	139	135	135
q7	942	609	593	593
q8	9218	2045	2074	2045
q9	8763	6549	6593	6549
q10	9095	3761	3751	3751
q11	446	236	239	236
q12	392	221	219	219
q13	17769	2976	3012	2976
q14	276	244	229	229
q15	527	490	481	481
q16	497	387	379	379
q17	958	640	689	640
q18	8006	7438	7383	7383
q19	4204	1445	1468	1445
q20	700	312	315	312
q21	4850	3134	3295	3134
q22	351	278	278	278
Total cold run time: 115001 ms
Total hot run time: 39674 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4363	4253	4245	4245
q2	371	258	268	258
q3	2999	2725	2875	2725
q4	1921	1669	1671	1669
q5	5592	5614	5544	5544
q6	228	138	132	132
q7	2174	1829	1836	1829
q8	3301	3433	3409	3409
q9	8804	8740	8887	8740
q10	4151	3958	3819	3819
q11	574	499	487	487
q12	807	646	664	646
q13	16909	3178	3185	3178
q14	312	296	292	292
q15	527	511	490	490
q16	488	472	422	422
q17	1808	1543	1541	1541
q18	8089	7962	7813	7813
q19	1732	1680	1567	1567
q20	2149	1889	1860	1860
q21	8957	4936	4803	4803
q22	547	483	511	483
Total cold run time: 76803 ms
Total hot run time: 55952 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173942 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5de8d0bb20c50032fe6158f1b26ae8463b498d4f, data reload: false

query1	912	367	373	367
query2	6467	1894	1904	1894
query3	6635	203	215	203
query4	27915	17406	17316	17316
query5	3684	472	474	472
query6	263	176	171	171
query7	4593	288	290	288
query8	238	205	191	191
query9	8499	2441	2418	2418
query10	452	288	261	261
query11	12135	10091	9972	9972
query12	117	91	83	83
query13	1635	380	396	380
query14	9410	7680	7824	7680
query15	223	164	164	164
query16	7231	320	316	316
query17	1796	594	544	544
query18	1819	289	278	278
query19	202	151	156	151
query20	90	88	84	84
query21	213	124	127	124
query22	4347	4120	4040	4040
query23	34356	33730	33704	33704
query24	10131	2960	2985	2960
query25	614	473	429	429
query26	715	158	155	155
query27	2160	279	289	279
query28	6283	2113	2097	2097
query29	924	669	661	661
query30	256	151	154	151
query31	972	750	777	750
query32	91	57	60	57
query33	695	339	319	319
query34	901	503	500	500
query35	700	629	623	623
query36	1151	960	991	960
query37	141	91	97	91
query38	2954	2814	2836	2814
query39	873	827	818	818
query40	207	121	128	121
query41	47	45	51	45
query42	120	107	100	100
query43	515	476	473	473
query44	1088	734	740	734
query45	195	167	165	165
query46	1098	750	725	725
query47	1853	1776	1768	1768
query48	359	307	303	303
query49	853	427	516	427
query50	787	391	392	391
query51	6877	6759	6715	6715
query52	110	94	96	94
query53	365	282	286	282
query54	883	441	448	441
query55	86	73	73	73
query56	289	261	269	261
query57	1158	1062	1069	1062
query58	246	253	264	253
query59	2825	2685	2593	2593
query60	289	281	298	281
query61	93	93	94	93
query62	801	626	654	626
query63	318	296	286	286
query64	9109	2209	1699	1699
query65	3185	3147	3121	3121
query66	754	333	329	329
query67	15588	15210	14925	14925
query68	8221	574	574	574
query69	804	420	343	343
query70	1186	1134	1108	1108
query71	523	281	279	279
query72	8465	5903	5477	5477
query73	1537	328	324	324
query74	6183	5660	5740	5660
query75	4649	2703	2699	2699
query76	4579	982	936	936
query77	787	301	313	301
query78	9721	9068	8871	8871
query79	8071	539	535	535
query80	2264	487	480	480
query81	599	227	222	222
query82	768	137	137	137
query83	285	186	165	165
query84	279	137	84	84
query85	1235	317	314	314
query86	390	279	293	279
query87	3306	3103	3080	3080
query88	4441	2400	2372	2372
query89	532	402	388	388
query90	2037	199	200	199
query91	132	105	104	104
query92	65	49	53	49
query93	6171	541	524	524
query94	1345	248	213	213
query95	413	325	326	325
query96	635	273	275	273
query97	3154	3035	3010	3010
query98	224	211	209	209
query99	1650	1277	1257	1257
Total cold run time: 297542 ms
Total hot run time: 173942 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.08 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5de8d0bb20c50032fe6158f1b26ae8463b498d4f, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.07
query5	0.49	0.49	0.49
query6	1.14	0.75	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.47	0.49
query10	0.55	0.53	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.58
query14	0.78	0.78	0.78
query15	0.84	0.81	0.82
query16	0.38	0.36	0.38
query17	1.05	1.01	1.00
query18	0.22	0.21	0.22
query19	1.89	1.67	1.77
query20	0.01	0.01	0.02
query21	15.39	0.75	0.66
query22	3.60	7.35	2.38
query23	18.28	1.40	1.26
query24	2.07	0.24	0.21
query25	0.15	0.08	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.22
query28	13.37	1.01	0.99
query29	12.58	3.34	3.32
query30	0.26	0.06	0.06
query31	2.87	0.39	0.38
query32	3.28	0.48	0.47
query33	2.91	2.91	2.94
query34	17.08	4.35	4.37
query35	4.42	4.44	4.41
query36	0.64	0.49	0.49
query37	0.19	0.15	0.17
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.26 s
Total hot run time: 31.08 s

wangbo
wangbo approved these changes Jul 16, 2024
View reviewed changes
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit a82aed8 into apache:master Jul 16, 2024
24 of 27 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@xinyiZzz @seawinde
[fix](arrow-flight-sql) Fix get Schema failed when `enable_parallel_r…
a41ebe4 
…esult_sink` is false (apache#37779)

if `enable_parallel_result_sink` is true, use queryID to get Schema and
fetch Result.
if `enable_parallel_result_sink` is false, use fragment 0 instance ID to
get Schema and fetch Result.

```
Caused by: cfjd.org.apache.arrow.flight.FlightRuntimeException: not found arrow flight schema, maybe query has been canceled, queryid: cef21a3bd35e40c5-bd25643a1e02dbb7
	at cfjd.org.apache.arrow.flight.CallStatus.toRuntimeException(CallStatus.java:131)
	at cfjd.org.apache.arrow.flight.grpc.StatusUtils.fromGrpcRuntimeException(StatusUtils.java:164)
	at cfjd.org.apache.arrow.flight.grpc.StatusUtils.fromThrowable(StatusUtils.java:185)
	at cfjd.org.apache.arrow.flight.FlightStream$Observer.onError(FlightStream.java:456)
```
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@xinyiZzz @dataroaring
[fix](arrow-flight-sql) Fix get Schema failed when `enable_parallel_r…
36ff00a 
…esult_sink` is false (#37779)

if `enable_parallel_result_sink` is true, use queryID to get Schema and
fetch Result.
if `enable_parallel_result_sink` is false, use fragment 0 instance ID to
get Schema and fetch Result.

```
Caused by: cfjd.org.apache.arrow.flight.FlightRuntimeException: not found arrow flight schema, maybe query has been canceled, queryid: cef21a3bd35e40c5-bd25643a1e02dbb7
	at cfjd.org.apache.arrow.flight.CallStatus.toRuntimeException(CallStatus.java:131)
	at cfjd.org.apache.arrow.flight.grpc.StatusUtils.fromGrpcRuntimeException(StatusUtils.java:164)
	at cfjd.org.apache.arrow.flight.grpc.StatusUtils.fromThrowable(StatusUtils.java:185)
	at cfjd.org.apache.arrow.flight.FlightStream$Observer.onError(FlightStream.java:456)
```
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wangbo wangbo wangbo approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@xinyiZzz @doris-robot @yiguolei @wangbo @dataroaring