Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](cloud) unify empty_partition_prune condition with nereids #34861

Merged
merged 3 commits into from
May 15, 2024

Conversation

freemandealer
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@freemandealer freemandealer marked this pull request as ready for review May 14, 2024 13:00
@freemandealer
Copy link
Contributor Author

run buildall

Copy link
Contributor

@gavinchou gavinchou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

TPC-H: Total hot run time: 41640 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fab1b218df494a37aa375d0a17e1f31e023c407b, data reload: false

------ Round 1 ----------------------------------
q1	17609	4363	4208	4208
q2	2018	188	184	184
q3	10461	1161	1234	1161
q4	10202	804	736	736
q5	7474	2731	2736	2731
q6	216	128	136	128
q7	1026	586	579	579
q8	9227	2124	2315	2124
q9	9358	6718	6720	6718
q10	9820	3890	3927	3890
q11	451	244	249	244
q12	493	233	228	228
q13	17314	3202	3170	3170
q14	279	231	211	211
q15	514	459	476	459
q16	496	386	390	386
q17	959	606	773	606
q18	8374	7898	7690	7690
q19	6446	1577	1557	1557
q20	643	333	303	303
q21	5303	4292	4049	4049
q22	373	278	284	278
Total cold run time: 119056 ms
Total hot run time: 41640 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4536	4462	4397	4397
q2	375	267	261	261
q3	3155	2888	2760	2760
q4	1900	1577	1635	1577
q5	5461	5479	5467	5467
q6	215	126	123	123
q7	2325	1959	2004	1959
q8	3239	3406	3476	3406
q9	8638	8723	8559	8559
q10	3953	3852	3852	3852
q11	605	500	485	485
q12	809	646	615	615
q13	17003	3103	3191	3103
q14	306	270	265	265
q15	523	473	472	472
q16	470	402	414	402
q17	1754	1499	1461	1461
q18	7671	7681	7498	7498
q19	1682	1557	1495	1495
q20	1973	1770	1795	1770
q21	5012	4975	4936	4936
q22	558	505	471	471
Total cold run time: 72163 ms
Total hot run time: 55334 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187917 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fab1b218df494a37aa375d0a17e1f31e023c407b, data reload: false

query1	908	384	366	366
query2	6437	2390	2317	2317
query3	6650	205	213	205
query4	25527	21141	21351	21141
query5	4119	411	420	411
query6	256	171	173	171
query7	4582	290	286	286
query8	239	184	194	184
query9	8562	2475	2467	2467
query10	438	255	251	251
query11	14701	14193	14154	14154
query12	135	91	86	86
query13	1649	373	380	373
query14	9935	7911	8403	7911
query15	242	167	172	167
query16	8027	260	261	260
query17	1784	546	540	540
query18	2047	270	268	268
query19	222	154	147	147
query20	89	84	81	81
query21	198	127	135	127
query22	5195	5006	5081	5006
query23	34267	33568	33638	33568
query24	10893	2860	2978	2860
query25	573	367	362	362
query26	707	158	153	153
query27	2173	320	338	320
query28	6007	2094	2073	2073
query29	842	596	603	596
query30	238	173	184	173
query31	974	746	760	746
query32	97	52	57	52
query33	727	251	242	242
query34	982	484	484	484
query35	812	666	679	666
query36	1035	907	921	907
query37	103	74	68	68
query38	2878	2756	2753	2753
query39	1599	1562	1563	1562
query40	203	124	122	122
query41	45	43	39	39
query42	103	94	96	94
query43	571	559	530	530
query44	1176	733	736	733
query45	267	247	247	247
query46	1093	735	720	720
query47	1957	1964	1889	1889
query48	375	304	296	296
query49	843	392	403	392
query50	777	386	409	386
query51	6778	6611	6605	6605
query52	106	88	94	88
query53	345	281	279	279
query54	841	434	428	428
query55	75	74	70	70
query56	240	227	223	223
query57	1236	1196	1169	1169
query58	219	208	227	208
query59	3343	3227	3031	3031
query60	251	240	234	234
query61	90	87	88	87
query62	653	492	469	469
query63	308	285	282	282
query64	8461	7438	7450	7438
query65	3169	3046	3081	3046
query66	781	348	346	346
query67	15260	15327	15194	15194
query68	4524	533	541	533
query69	474	300	301	300
query70	1173	1076	1170	1076
query71	368	264	266	264
query72	7223	2575	2352	2352
query73	700	334	334	334
query74	6539	6181	6248	6181
query75	3325	2616	2609	2609
query76	2313	1040	958	958
query77	400	273	265	265
query78	10641	10151	10168	10151
query79	1481	511	522	511
query80	1011	451	447	447
query81	531	251	243	243
query82	967	98	97	97
query83	240	164	167	164
query84	261	85	83	83
query85	1136	268	258	258
query86	391	325	304	304
query87	3295	3179	3146	3146
query88	3269	2449	2440	2440
query89	469	387	392	387
query90	1887	182	188	182
query91	128	96	98	96
query92	55	46	50	46
query93	1021	514	493	493
query94	1140	183	188	183
query95	390	307	295	295
query96	581	268	278	268
query97	3162	2994	3031	2994
query98	237	228	220	220
query99	1181	889	888	888
Total cold run time: 276928 ms
Total hot run time: 187917 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fab1b218df494a37aa375d0a17e1f31e023c407b, data reload: false

query1	0.03	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.09	0.08
query5	0.49	0.50	0.51
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.06	0.04	0.05
query9	0.54	0.49	0.49
query10	0.54	0.54	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.60	0.60
query14	0.77	0.78	0.77
query15	0.82	0.81	0.81
query16	0.38	0.37	0.38
query17	1.01	1.04	1.01
query18	0.22	0.27	0.22
query19	1.77	1.79	1.80
query20	0.01	0.01	0.01
query21	15.44	0.69	0.68
query22	4.54	6.21	2.38
query23	18.33	1.36	1.29
query24	1.57	0.27	0.23
query25	0.14	0.09	0.08
query26	0.25	0.17	0.16
query27	0.08	0.08	0.07
query28	13.46	1.01	0.99
query29	13.24	3.30	3.33
query30	0.24	0.05	0.06
query31	2.89	0.38	0.38
query32	3.28	0.46	0.45
query33	2.79	2.84	2.84
query34	17.17	4.41	4.45
query35	4.48	4.48	4.64
query36	0.65	0.45	0.46
query37	0.17	0.15	0.15
query38	0.16	0.14	0.15
query39	0.05	0.03	0.04
query40	0.16	0.15	0.13
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.03	0.04	0.03
Total cold run time: 109.95 s
Total hot run time: 31.12 s

@hello-stephen
Copy link
Contributor

run cloud_p0

Signed-off-by: freemandealer <[email protected]>
@freemandealer
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39584 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit af97e3b9fc8b6354b74c8b3fcedd1754ccf3947b, data reload: false

------ Round 1 ----------------------------------
q1	18162	4375	4233	4233
q2	2014	192	195	192
q3	10494	1123	1236	1123
q4	10160	846	835	835
q5	7450	2648	2569	2569
q6	212	133	129	129
q7	938	535	503	503
q8	9226	2060	2064	2060
q9	8877	6543	6467	6467
q10	8893	3668	3676	3668
q11	484	243	231	231
q12	401	226	216	216
q13	17782	3004	2927	2927
q14	260	220	217	217
q15	499	474	470	470
q16	521	404	375	375
q17	972	635	712	635
q18	8216	7534	7374	7374
q19	6820	1533	1490	1490
q20	694	319	302	302
q21	5000	3291	4031	3291
q22	344	277	279	277
Total cold run time: 118419 ms
Total hot run time: 39584 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4444	4225	4226	4225
q2	372	267	264	264
q3	2976	2772	2754	2754
q4	1882	1618	1631	1618
q5	5275	5256	5288	5256
q6	209	124	136	124
q7	1734	1377	1366	1366
q8	3167	3336	3348	3336
q9	8330	8325	8339	8325
q10	3919	3706	3704	3704
q11	589	483	483	483
q12	753	620	597	597
q13	16518	2936	2990	2936
q14	290	272	261	261
q15	525	483	485	483
q16	473	430	416	416
q17	1786	1456	1481	1456
q18	7653	7545	7408	7408
q19	1651	1540	1563	1540
q20	1940	1775	1746	1746
q21	5087	4934	4949	4934
q22	563	507	497	497
Total cold run time: 70136 ms
Total hot run time: 53729 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186995 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit af97e3b9fc8b6354b74c8b3fcedd1754ccf3947b, data reload: false

query1	924	403	366	366
query2	6450	2448	2210	2210
query3	6670	211	215	211
query4	24172	21209	21181	21181
query5	4208	415	423	415
query6	271	172	176	172
query7	4585	293	293	293
query8	240	185	179	179
query9	8529	2440	2404	2404
query10	433	257	256	256
query11	15076	14278	14185	14185
query12	143	94	90	90
query13	1651	387	368	368
query14	10622	6800	6964	6800
query15	229	176	177	176
query16	7968	270	267	267
query17	1849	577	564	564
query18	2015	294	284	284
query19	218	160	150	150
query20	101	86	87	86
query21	197	136	135	135
query22	5011	4808	4861	4808
query23	34331	33725	33810	33725
query24	11690	2950	2858	2858
query25	674	370	372	370
query26	1776	160	157	157
query27	2994	323	324	323
query28	7417	2077	2064	2064
query29	1089	606	631	606
query30	309	177	180	177
query31	994	759	740	740
query32	91	51	57	51
query33	748	259	247	247
query34	1083	483	493	483
query35	807	675	673	673
query36	1064	925	907	907
query37	275	73	70	70
query38	2917	2777	2790	2777
query39	1605	1558	1570	1558
query40	274	126	123	123
query41	46	42	43	42
query42	100	95	96	95
query43	592	547	560	547
query44	1175	730	730	730
query45	271	246	246	246
query46	1082	711	711	711
query47	1985	1894	1878	1878
query48	401	296	288	288
query49	1204	398	404	398
query50	770	384	396	384
query51	6948	6700	6784	6700
query52	108	92	95	92
query53	351	286	278	278
query54	913	446	434	434
query55	73	75	71	71
query56	240	226	223	223
query57	1255	1135	1140	1135
query58	218	231	200	200
query59	3427	3193	3095	3095
query60	272	236	240	236
query61	95	113	90	90
query62	680	470	488	470
query63	306	279	294	279
query64	9717	7432	7364	7364
query65	3162	3137	3159	3137
query66	1373	358	338	338
query67	15335	15414	15226	15226
query68	5745	545	543	543
query69	539	303	325	303
query70	1220	1149	1158	1149
query71	460	271	268	268
query72	7662	2616	2346	2346
query73	728	330	325	325
query74	6660	6196	6202	6196
query75	3888	2632	2659	2632
query76	3613	915	921	915
query77	606	262	266	262
query78	10625	10124	10128	10124
query79	2722	519	504	504
query80	1260	440	450	440
query81	508	242	249	242
query82	1044	106	101	101
query83	193	167	169	167
query84	265	85	88	85
query85	1384	321	269	269
query86	462	347	334	334
query87	3301	3136	3095	3095
query88	4309	2412	2416	2412
query89	480	381	397	381
query90	1961	192	195	192
query91	145	101	97	97
query92	63	52	49	49
query93	3010	519	497	497
query94	1170	179	185	179
query95	406	311	308	308
query96	612	275	270	270
query97	3167	2987	2990	2987
query98	239	218	215	215
query99	1288	930	912	912
Total cold run time: 293534 ms
Total hot run time: 186995 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit af97e3b9fc8b6354b74c8b3fcedd1754ccf3947b, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.04	0.04
query3	0.23	0.04	0.06
query4	1.66	0.08	0.08
query5	0.49	0.49	0.52
query6	1.13	0.71	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.53	0.48	0.49
query10	0.54	0.55	0.53
query11	0.16	0.11	0.12
query12	0.14	0.12	0.11
query13	0.59	0.58	0.61
query14	0.77	0.78	0.78
query15	0.84	0.81	0.81
query16	0.37	0.36	0.37
query17	1.00	0.96	1.01
query18	0.23	0.25	0.25
query19	1.81	1.68	1.70
query20	0.02	0.01	0.01
query21	15.71	0.66	0.64
query22	4.29	7.71	1.71
query23	18.28	1.44	1.28
query24	1.52	0.31	0.22
query25	0.15	0.08	0.08
query26	0.26	0.16	0.16
query27	0.07	0.07	0.09
query28	13.39	1.02	1.00
query29	13.21	3.30	3.26
query30	0.24	0.06	0.06
query31	2.95	0.38	0.39
query32	3.83	0.48	0.47
query33	2.82	2.80	2.82
query34	17.16	4.38	4.45
query35	4.50	4.49	4.52
query36	0.63	0.46	0.47
query37	0.17	0.16	0.15
query38	0.15	0.15	0.14
query39	0.05	0.04	0.03
query40	0.16	0.13	0.14
query41	0.09	0.05	0.04
query42	0.06	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 110.43 s
Total hot run time: 30.12 s

@freemandealer
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41642 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 20f84debf9a3ec476dfe0951841b46ca75d93ff9, data reload: false

------ Round 1 ----------------------------------
q1	6844	4257	4261	4257
q2	624	191	202	191
q3	1377	1217	1206	1206
q4	1007	773	821	773
q5	2629	2714	2646	2646
q6	220	140	139	139
q7	986	586	581	581
q8	1956	2202	2287	2202
q9	6950	6835	6687	6687
q10	4014	3946	3908	3908
q11	366	246	244	244
q12	388	228	246	228
q13	16356	2961	3126	2961
q14	269	219	229	219
q15	535	472	489	472
q16	504	393	388	388
q17	972	715	650	650
q18	8307	7783	7896	7783
q19	1661	1569	1456	1456
q20	548	315	316	315
q21	5245	4048	4204	4048
q22	355	289	288	288
Total cold run time: 62113 ms
Total hot run time: 41642 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4386	4353	4360	4353
q2	385	276	271	271
q3	3189	2906	2780	2780
q4	1906	1578	1603	1578
q5	5271	5292	5316	5292
q6	216	124	125	124
q7	2271	1896	1887	1887
q8	3183	3360	3340	3340
q9	8397	8379	8354	8354
q10	3887	3735	3679	3679
q11	574	491	491	491
q12	770	633	584	584
q13	5030	3002	2993	2993
q14	282	261	255	255
q15	519	464	467	464
q16	465	415	410	410
q17	1765	1465	1495	1465
q18	7639	7658	7475	7475
q19	1699	1575	1586	1575
q20	1993	1755	1758	1755
q21	4898	5020	4814	4814
q22	559	485	474	474
Total cold run time: 59284 ms
Total hot run time: 54413 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187503 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 20f84debf9a3ec476dfe0951841b46ca75d93ff9, data reload: false

query1	905	382	366	366
query2	6434	2380	2355	2355
query3	6628	212	214	212
query4	25545	21375	21172	21172
query5	4126	417	434	417
query6	279	174	174	174
query7	4587	297	285	285
query8	251	217	191	191
query9	8668	2407	2400	2400
query10	437	250	271	250
query11	15210	14213	14171	14171
query12	125	93	92	92
query13	1647	382	383	382
query14	9845	8013	7804	7804
query15	236	175	175	175
query16	7646	267	271	267
query17	1283	595	569	569
query18	1951	289	289	289
query19	212	159	156	156
query20	96	88	87	87
query21	197	132	132	132
query22	5084	4876	4844	4844
query23	34030	33612	33531	33531
query24	5039	2851	2931	2851
query25	478	390	383	383
query26	690	156	154	154
query27	1846	328	328	328
query28	3663	2088	2056	2056
query29	822	604	600	600
query30	232	173	175	173
query31	933	771	734	734
query32	59	57	58	57
query33	453	246	254	246
query34	870	495	509	495
query35	787	673	664	664
query36	1056	914	937	914
query37	108	70	71	70
query38	2850	2776	2729	2729
query39	1598	1566	1562	1562
query40	195	126	125	125
query41	45	40	39	39
query42	105	97	96	96
query43	593	540	556	540
query44	1062	721	729	721
query45	268	253	257	253
query46	1058	740	716	716
query47	1965	1870	1909	1870
query48	361	293	299	293
query49	768	396	404	396
query50	756	384	400	384
query51	6848	6657	6684	6657
query52	116	88	91	88
query53	358	293	281	281
query54	533	435	458	435
query55	78	71	72	71
query56	246	222	227	222
query57	1239	1147	1180	1147
query58	221	207	207	207
query59	3428	3253	3089	3089
query60	262	235	236	235
query61	89	86	86	86
query62	574	490	470	470
query63	319	282	286	282
query64	7741	7461	7336	7336
query65	3131	3063	3129	3063
query66	730	349	335	335
query67	15432	15496	14971	14971
query68	4588	551	535	535
query69	516	302	311	302
query70	1135	1140	1146	1140
query71	452	273	286	273
query72	8165	2545	2408	2408
query73	711	327	328	327
query74	6527	6190	6182	6182
query75	3423	2704	2614	2614
query76	2601	977	971	971
query77	591	262	265	262
query78	10606	10327	10022	10022
query79	1783	530	516	516
query80	864	442	445	442
query81	504	246	248	246
query82	668	100	94	94
query83	231	162	163	162
query84	246	83	85	83
query85	1097	269	266	266
query86	464	295	296	295
query87	3266	3133	3141	3133
query88	4188	2401	2411	2401
query89	466	395	451	395
query90	2046	189	189	189
query91	127	99	102	99
query92	62	47	50	47
query93	2570	518	493	493
query94	1170	183	191	183
query95	396	313	305	305
query96	600	275	273	273
query97	3134	2994	3005	2994
query98	235	226	209	209
query99	1255	924	897	897
Total cold run time: 270114 ms
Total hot run time: 187503 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.91 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 20f84debf9a3ec476dfe0951841b46ca75d93ff9, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.22	0.06	0.05
query4	1.65	0.08	0.08
query5	0.50	0.49	0.51
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.54	0.56	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.60	0.62
query14	0.78	0.77	0.80
query15	0.83	0.81	0.82
query16	0.37	0.36	0.36
query17	0.98	1.00	0.98
query18	0.21	0.24	0.26
query19	1.78	1.71	1.67
query20	0.02	0.01	0.00
query21	15.51	0.68	0.64
query22	4.23	5.80	3.26
query23	18.25	1.37	1.32
query24	2.00	0.22	0.21
query25	0.14	0.09	0.08
query26	0.27	0.16	0.16
query27	0.09	0.08	0.08
query28	13.39	1.02	1.00
query29	13.52	3.39	3.37
query30	0.23	0.06	0.05
query31	2.87	0.39	0.39
query32	3.28	0.46	0.46
query33	2.84	2.81	2.84
query34	17.04	4.40	4.44
query35	4.52	4.52	4.55
query36	0.68	0.47	0.49
query37	0.17	0.14	0.16
query38	0.15	0.14	0.14
query39	0.05	0.03	0.04
query40	0.16	0.13	0.14
query41	0.09	0.05	0.05
query42	0.05	0.04	0.04
query43	0.03	0.04	0.03
Total cold run time: 110.18 s
Total hot run time: 31.91 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 15, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 2b11827 into apache:master May 15, 2024
26 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants