Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](Nereids) use 1 as narrowest column when do column pruning on union #41719

Merged
merged 1 commit into from
Oct 14, 2024

Conversation

morrySnow
Copy link
Contributor

just like previous PR #41548

this PR process union node to ensure not require any column from its children when it is required by its parent with empty slot set

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40860 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01c7694e53fd85972b213a8363834872c7ca77c8, data reload: false

------ Round 1 ----------------------------------
q1	17797	7947	7257	7257
q2	2024	307	290	290
q3	11707	1066	1193	1066
q4	10588	742	676	676
q5	7739	2857	2823	2823
q6	237	150	147	147
q7	1004	607	634	607
q8	9355	1907	1972	1907
q9	6510	6469	6393	6393
q10	7010	2303	2296	2296
q11	440	249	248	248
q12	406	211	212	211
q13	17775	2997	2980	2980
q14	255	216	215	215
q15	569	519	507	507
q16	635	577	589	577
q17	979	521	531	521
q18	7276	6792	6803	6792
q19	1352	1020	953	953
q20	474	202	195	195
q21	4079	3241	3205	3205
q22	1101	1022	994	994
Total cold run time: 109312 ms
Total hot run time: 40860 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7308	7208	7224	7208
q2	325	228	230	228
q3	2976	2983	2925	2925
q4	2071	1859	1828	1828
q5	5770	5733	5713	5713
q6	221	154	142	142
q7	2245	1866	1828	1828
q8	3366	3629	3385	3385
q9	8997	8911	8844	8844
q10	3580	3512	3486	3486
q11	601	480	487	480
q12	827	644	663	644
q13	9403	3183	3116	3116
q14	303	281	300	281
q15	553	536	525	525
q16	686	638	672	638
q17	1858	1728	1614	1614
q18	8344	7768	7517	7517
q19	1728	1571	1268	1268
q20	2127	1864	1854	1854
q21	5509	5372	5447	5372
q22	1116	1060	1080	1060
Total cold run time: 69914 ms
Total hot run time: 59956 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191631 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01c7694e53fd85972b213a8363834872c7ca77c8, data reload: false

query1	924	389	417	389
query2	6253	2093	2050	2050
query3	8681	191	199	191
query4	34153	23510	23362	23362
query5	3843	479	470	470
query6	277	166	165	165
query7	4199	303	306	303
query8	281	215	220	215
query9	9503	2738	2687	2687
query10	465	285	279	279
query11	17828	15142	15138	15138
query12	153	99	103	99
query13	1585	472	460	460
query14	9215	7338	7267	7267
query15	263	167	175	167
query16	8074	462	455	455
query17	1687	627	601	601
query18	2167	310	316	310
query19	359	160	158	158
query20	123	118	111	111
query21	221	108	112	108
query22	4960	4695	4740	4695
query23	34623	34316	33928	33928
query24	11033	2904	2812	2812
query25	647	402	397	397
query26	1303	165	169	165
query27	2628	301	306	301
query28	7784	2436	2423	2423
query29	887	438	435	435
query30	251	150	157	150
query31	994	797	770	770
query32	92	53	56	53
query33	766	297	294	294
query34	912	495	504	495
query35	866	730	728	728
query36	1081	926	951	926
query37	151	84	84	84
query38	3993	3905	3819	3819
query39	1504	1538	1401	1401
query40	249	95	95	95
query41	47	44	45	44
query42	114	95	96	95
query43	515	486	492	486
query44	1276	812	822	812
query45	193	161	168	161
query46	1150	735	718	718
query47	1978	1865	1871	1865
query48	440	358	350	350
query49	897	410	403	403
query50	810	429	419	419
query51	7136	6888	6936	6888
query52	100	87	98	87
query53	258	182	183	182
query54	1187	477	480	477
query55	79	79	78	78
query56	297	273	276	273
query57	1313	1181	1166	1166
query58	241	220	255	220
query59	3230	3060	3021	3021
query60	297	285	270	270
query61	104	104	103	103
query62	883	668	670	668
query63	223	184	183	183
query64	4957	623	599	599
query65	3311	3133	3193	3133
query66	1391	298	302	298
query67	15942	15727	15479	15479
query68	4377	578	564	564
query69	468	297	297	297
query70	1176	1121	1130	1121
query71	343	275	318	275
query72	7165	3939	3933	3933
query73	761	349	343	343
query74	10227	8936	8995	8936
query75	3398	2665	2660	2660
query76	2852	940	937	937
query77	579	304	278	278
query78	10453	9541	9502	9502
query79	2120	590	589	589
query80	1255	446	452	446
query81	575	239	234	234
query82	700	142	136	136
query83	293	150	134	134
query84	278	81	75	75
query85	1406	311	296	296
query86	393	289	304	289
query87	4589	4232	4345	4232
query88	3128	2417	2371	2371
query89	406	282	278	278
query90	1913	184	181	181
query91	146	109	101	101
query92	64	49	48	48
query93	2148	546	548	546
query94	989	288	298	288
query95	354	252	244	244
query96	609	278	282	278
query97	3248	3110	3146	3110
query98	209	200	194	194
query99	1626	1304	1281	1281
Total cold run time: 301595 ms
Total hot run time: 191631 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.55 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 01c7694e53fd85972b213a8363834872c7ca77c8, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.52	0.51	0.53
query6	1.13	0.74	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.55	0.52	0.48
query10	0.56	0.53	0.56
query11	0.14	0.11	0.11
query12	0.14	0.10	0.10
query13	0.62	0.59	0.60
query14	2.73	2.76	2.71
query15	0.91	0.83	0.84
query16	0.38	0.39	0.39
query17	1.04	1.04	1.05
query18	0.20	0.20	0.20
query19	1.93	1.77	2.06
query20	0.01	0.01	0.01
query21	15.35	0.61	0.60
query22	2.60	2.08	1.89
query23	16.79	0.84	0.80
query24	2.83	1.31	0.81
query25	0.33	0.10	0.04
query26	0.53	0.13	0.13
query27	0.04	0.07	0.04
query28	10.85	1.08	1.06
query29	12.50	3.24	3.28
query30	0.24	0.06	0.06
query31	2.87	0.38	0.37
query32	3.28	0.46	0.45
query33	2.99	3.06	3.05
query34	16.78	4.52	4.53
query35	4.57	4.47	4.53
query36	0.69	0.49	0.51
query37	0.08	0.05	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.14	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.58 s
Total hot run time: 32.55 s

just like previous PR apache#41548

this PR process union node to ensure not require any column from its
children when it is required by its parent with empty slot set
@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41158 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 82b0504148afc73baa460a5e0fba91980a889296, data reload: false

------ Round 1 ----------------------------------
q1	17620	7467	7258	7258
q2	2019	293	293	293
q3	12102	1067	1173	1067
q4	10578	737	733	733
q5	7795	2901	2826	2826
q6	243	161	159	159
q7	1040	657	668	657
q8	9368	1937	1936	1936
q9	6564	6440	6418	6418
q10	6965	2373	2365	2365
q11	445	253	255	253
q12	416	221	211	211
q13	17757	2953	2999	2953
q14	246	216	210	210
q15	576	523	523	523
q16	636	581	578	578
q17	973	624	527	527
q18	7293	6732	6878	6732
q19	1351	989	986	986
q20	494	210	201	201
q21	3952	3298	3320	3298
q22	1067	974	977	974
Total cold run time: 109500 ms
Total hot run time: 41158 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7653	7171	7177	7171
q2	328	227	223	223
q3	3035	2930	2962	2930
q4	2150	1827	1807	1807
q5	5724	5774	5710	5710
q6	241	152	146	146
q7	2203	1845	1815	1815
q8	3347	3601	3412	3412
q9	8947	8917	8893	8893
q10	3599	3582	3512	3512
q11	598	493	493	493
q12	845	642	638	638
q13	9285	3133	3186	3133
q14	293	270	272	270
q15	564	535	529	529
q16	680	641	633	633
q17	1862	1625	1624	1624
q18	8349	7762	7673	7673
q19	1726	1411	1434	1411
q20	2127	1903	1927	1903
q21	5646	5489	5528	5489
q22	1141	1080	1064	1064
Total cold run time: 70343 ms
Total hot run time: 60479 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191590 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 82b0504148afc73baa460a5e0fba91980a889296, data reload: false

query1	952	389	394	389
query2	6255	2140	2147	2140
query3	8679	191	200	191
query4	34038	23611	23383	23383
query5	3484	502	464	464
query6	265	164	170	164
query7	4192	304	301	301
query8	277	214	223	214
query9	9519	2692	2691	2691
query10	454	291	275	275
query11	17973	15253	15221	15221
query12	145	105	95	95
query13	1582	477	444	444
query14	9717	6862	7005	6862
query15	253	168	174	168
query16	7895	487	460	460
query17	1614	625	607	607
query18	2076	313	338	313
query19	255	155	162	155
query20	115	111	108	108
query21	213	107	105	105
query22	4896	4535	4559	4535
query23	34960	33867	33957	33867
query24	11059	2770	2730	2730
query25	581	416	402	402
query26	1058	157	163	157
query27	2146	300	300	300
query28	6516	2430	2407	2407
query29	811	438	440	438
query30	297	149	156	149
query31	1036	817	800	800
query32	93	54	56	54
query33	809	289	301	289
query34	931	514	506	506
query35	901	741	755	741
query36	1141	939	946	939
query37	151	89	95	89
query38	4018	3896	3841	3841
query39	1494	1454	1397	1397
query40	200	96	97	96
query41	48	44	44	44
query42	114	94	97	94
query43	543	509	499	499
query44	1250	816	802	802
query45	194	164	166	164
query46	1134	718	715	715
query47	1971	1858	1896	1858
query48	433	352	348	348
query49	978	418	409	409
query50	835	429	426	426
query51	7175	6985	6927	6927
query52	97	86	89	86
query53	269	186	183	183
query54	1235	479	487	479
query55	84	78	79	78
query56	274	264	261	261
query57	1270	1171	1138	1138
query58	233	229	251	229
query59	3207	3222	3116	3116
query60	296	269	255	255
query61	107	104	101	101
query62	863	673	639	639
query63	220	185	183	183
query64	3760	622	598	598
query65	3223	3191	3165	3165
query66	829	310	299	299
query67	16001	15729	15690	15690
query68	4520	562	554	554
query69	487	322	302	302
query70	1231	1144	1103	1103
query71	358	275	267	267
query72	7303	4017	3965	3965
query73	769	351	355	351
query74	10084	8960	8990	8960
query75	3461	2678	2667	2667
query76	2841	899	944	899
query77	623	292	290	290
query78	10542	9628	9502	9502
query79	3387	614	600	600
query80	1455	447	448	447
query81	577	239	241	239
query82	700	142	141	141
query83	288	138	132	132
query84	273	81	82	81
query85	1268	291	300	291
query86	389	298	299	298
query87	4561	4234	4383	4234
query88	4092	2450	2412	2412
query89	417	286	303	286
query90	1884	193	190	190
query91	139	109	107	107
query92	64	50	48	48
query93	2179	561	545	545
query94	936	288	267	267
query95	359	257	254	254
query96	606	289	283	283
query97	3212	3130	3119	3119
query98	211	203	191	191
query99	1558	1324	1292	1292
Total cold run time: 300468 ms
Total hot run time: 191590 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.57 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 82b0504148afc73baa460a5e0fba91980a889296, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.50	0.48	0.49
query6	1.14	0.71	0.74
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.55	0.51	0.49
query10	0.55	0.55	0.53
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.59	0.60
query14	2.73	2.73	2.77
query15	0.90	0.83	0.84
query16	0.38	0.39	0.39
query17	1.07	1.04	1.02
query18	0.20	0.20	0.20
query19	1.94	1.88	1.96
query20	0.02	0.01	0.01
query21	15.36	0.61	0.58
query22	2.67	2.52	1.94
query23	16.93	0.85	0.86
query24	3.33	0.77	1.99
query25	0.33	0.04	0.09
query26	0.57	0.14	0.13
query27	0.03	0.04	0.05
query28	10.10	1.09	1.09
query29	12.58	3.26	3.24
query30	0.25	0.06	0.06
query31	2.87	0.39	0.37
query32	3.29	0.46	0.47
query33	3.00	2.99	3.00
query34	16.79	4.47	4.42
query35	4.48	4.53	4.48
query36	0.66	0.48	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.61 s
Total hot run time: 32.57 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 14, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 5ebfed9 into apache:master Oct 14, 2024
26 of 27 checks passed
@morrySnow morrySnow deleted the opt_union_prune branch October 14, 2024 03:21
qzsee pushed a commit to qzsee/incubator-doris that referenced this pull request Oct 16, 2024
…ion (apache#41719)

just like previous PR apache#41548

this PR process union node to ensure not require any column from its
children when it is required by its parent with empty slot set
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Oct 16, 2024
…ion (apache#41719)

pick from master apache#41719

just like previous PR apache#41548

this PR process union node to ensure not require any column from its
children when it is required by its parent with empty slot set
morrySnow added a commit that referenced this pull request Oct 17, 2024
…ion (#41719) (#41975)

pick from master #41719

just like previous PR #41548

this PR process union node to ensure not require any column from its
children when it is required by its parent with empty slot set
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Oct 29, 2024
…ion (apache#41719)

pick from master apache#41719

just like previous PR apache#41548

this PR process union node to ensure not require any column from its
children when it is required by its parent with empty slot set
morrySnow added a commit that referenced this pull request Oct 29, 2024
…ion (#41719) (#42860)

pick from master #41719

just like previous PR #41548

this PR process union node to ensure not require any column from its
children when it is required by its parent with empty slot set
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants