Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor](exec) refactor the partion sort code #46262

Merged
merged 2 commits into from
Jan 3, 2025

Conversation

HappenLee
Copy link
Contributor

What problem does this PR solve?

do refactor the partion sort code remove the unless code

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@HappenLee
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

TPC-H: Total hot run time: 32508 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 360fc227c357da89b3434b3d4c431b06cf89e054, data reload: false

------ Round 1 ----------------------------------
q1	17600	6136	6032	6032
q2	2046	312	176	176
q3	10407	1254	701	701
q4	10220	849	427	427
q5	7519	2238	1977	1977
q6	219	184	145	145
q7	907	761	614	614
q8	9225	1369	1149	1149
q9	5183	4925	4898	4898
q10	6767	2309	1861	1861
q11	475	281	266	266
q12	342	355	222	222
q13	17749	3570	2964	2964
q14	241	235	230	230
q15	569	494	497	494
q16	626	614	591	591
q17	582	852	335	335
q18	7189	6425	6524	6425
q19	1239	968	552	552
q20	311	314	196	196
q21	2956	2163	1949	1949
q22	368	324	304	304
Total cold run time: 102740 ms
Total hot run time: 32508 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6213	6239	6184	6184
q2	239	334	227	227
q3	2240	2687	2323	2323
q4	1423	1808	1373	1373
q5	4372	4742	4731	4731
q6	186	180	142	142
q7	2085	2008	1822	1822
q8	2592	2811	2637	2637
q9	7365	7243	7198	7198
q10	3049	3325	2837	2837
q11	593	509	496	496
q12	668	763	626	626
q13	3417	3763	3039	3039
q14	307	321	272	272
q15	572	515	494	494
q16	647	677	649	649
q17	1186	1740	1231	1231
q18	7760	7306	7277	7277
q19	805	1148	1011	1011
q20	1936	1959	1795	1795
q21	5386	5030	4827	4827
q22	605	615	545	545
Total cold run time: 53646 ms
Total hot run time: 51736 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189377 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 360fc227c357da89b3434b3d4c431b06cf89e054, data reload: false

query1	979	381	364	364
query2	6530	2461	2362	2362
query3	6709	216	208	208
query4	33665	23567	23570	23567
query5	4363	624	461	461
query6	318	228	184	184
query7	4621	504	298	298
query8	303	252	239	239
query9	9416	2666	2654	2654
query10	479	309	258	258
query11	18182	15583	15192	15192
query12	158	109	107	107
query13	1701	542	422	422
query14	11396	6745	7168	6745
query15	251	198	189	189
query16	7940	613	497	497
query17	1560	756	584	584
query18	2100	428	295	295
query19	214	181	154	154
query20	123	114	112	112
query21	214	153	105	105
query22	4225	4420	4123	4123
query23	34306	33025	33441	33025
query24	6410	2344	2236	2236
query25	471	460	376	376
query26	1207	273	153	153
query27	1999	451	332	332
query28	5148	2455	2421	2421
query29	695	534	410	410
query30	227	185	153	153
query31	999	922	818	818
query32	68	65	60	60
query33	490	353	297	297
query34	740	819	499	499
query35	784	849	725	725
query36	1020	1057	956	956
query37	111	108	74	74
query38	4101	4286	4056	4056
query39	1478	1426	1454	1426
query40	205	114	101	101
query41	47	54	44	44
query42	125	100	101	100
query43	514	523	473	473
query44	1302	819	820	819
query45	178	203	166	166
query46	859	1033	652	652
query47	1942	1944	1847	1847
query48	377	405	331	331
query49	773	468	381	381
query50	630	646	388	388
query51	7133	7279	7035	7035
query52	102	102	90	90
query53	225	265	182	182
query54	476	483	409	409
query55	80	78	78	78
query56	267	262	239	239
query57	1198	1179	1131	1131
query58	231	228	243	228
query59	3124	3100	2958	2958
query60	282	262	256	256
query61	114	140	106	106
query62	843	823	735	735
query63	227	188	199	188
query64	4136	969	643	643
query65	3323	3178	3247	3178
query66	1052	416	306	306
query67	15916	16077	15578	15578
query68	9114	750	510	510
query69	465	286	254	254
query70	1205	1086	1106	1086
query71	441	276	254	254
query72	5863	3845	3841	3841
query73	661	755	371	371
query74	9907	9041	8933	8933
query75	4654	3154	2670	2670
query76	5399	1178	840	840
query77	971	350	277	277
query78	10113	10226	9418	9418
query79	2729	874	605	605
query80	701	519	430	430
query81	481	273	233	233
query82	460	140	129	129
query83	193	160	142	142
query84	284	98	68	68
query85	844	370	301	301
query86	350	330	313	313
query87	4402	4407	4504	4407
query88	2957	2220	2180	2180
query89	406	339	296	296
query90	1958	189	185	185
query91	134	134	105	105
query92	70	62	55	55
query93	977	865	540	540
query94	663	399	293	293
query95	335	260	253	253
query96	478	600	283	283
query97	2713	2803	2705	2705
query98	223	206	196	196
query99	1637	1570	1425	1425
Total cold run time: 293170 ms
Total hot run time: 189377 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.47 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 360fc227c357da89b3434b3d4c431b06cf89e054, data reload: false

query1	0.04	0.04	0.05
query2	0.07	0.04	0.04
query3	0.24	0.07	0.07
query4	1.61	0.11	0.11
query5	0.42	0.41	0.40
query6	1.15	0.65	0.64
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.58	0.50	0.49
query10	0.55	0.56	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.10
query13	0.60	0.61	0.60
query14	2.73	2.82	2.76
query15	0.90	0.82	0.82
query16	0.38	0.39	0.38
query17	1.08	1.02	1.05
query18	0.23	0.20	0.21
query19	1.83	1.86	2.04
query20	0.01	0.01	0.01
query21	15.39	0.95	0.59
query22	0.77	0.75	0.75
query23	15.29	1.47	0.54
query24	3.15	1.02	2.04
query25	0.16	0.15	0.08
query26	0.22	0.15	0.15
query27	0.06	0.08	0.05
query28	14.35	1.49	1.05
query29	12.64	3.92	3.26
query30	0.25	0.09	0.07
query31	2.83	0.58	0.38
query32	3.24	0.54	0.45
query33	3.11	3.10	3.19
query34	16.59	5.14	4.50
query35	4.47	4.44	4.52
query36	0.68	0.51	0.47
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.47 s
Total hot run time: 31.47 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.90% (10122/26021)
Line Coverage: 29.90% (85555/286134)
Region Coverage: 29.01% (43704/150631)
Branch Coverage: 25.55% (22307/87312)
Coverage Report: http://coverage.selectdb-in.cc/coverage/360fc227c357da89b3434b3d4c431b06cf89e054_360fc227c357da89b3434b3d4c431b06cf89e054/report/index.html

@HappenLee
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32740 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f975aac0978f266e866546912c4d009f268593b, data reload: false

------ Round 1 ----------------------------------
q1	17582	6204	6054	6054
q2	2041	330	174	174
q3	10389	1262	722	722
q4	10221	897	445	445
q5	7504	2199	2000	2000
q6	210	184	148	148
q7	897	758	619	619
q8	9228	1378	1197	1197
q9	5276	4946	4995	4946
q10	6772	2320	1856	1856
q11	477	283	263	263
q12	348	361	218	218
q13	17781	3650	3084	3084
q14	248	231	216	216
q15	553	526	488	488
q16	633	630	585	585
q17	593	849	339	339
q18	7245	6590	6382	6382
q19	2605	977	553	553
q20	306	314	187	187
q21	2817	2159	1957	1957
q22	356	330	307	307
Total cold run time: 104082 ms
Total hot run time: 32740 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6417	6282	6231	6231
q2	232	327	230	230
q3	2247	2664	2362	2362
q4	1384	1855	1390	1390
q5	4366	4749	4912	4749
q6	195	179	143	143
q7	2032	1991	1810	1810
q8	2622	2778	2743	2743
q9	7332	7319	7234	7234
q10	3066	3305	2825	2825
q11	598	513	496	496
q12	653	739	641	641
q13	3545	3864	3268	3268
q14	288	312	286	286
q15	559	506	503	503
q16	672	684	652	652
q17	1218	1754	1257	1257
q18	7772	7544	7133	7133
q19	853	1140	1038	1038
q20	1964	1949	1811	1811
q21	5511	5033	4946	4946
q22	603	632	592	592
Total cold run time: 54129 ms
Total hot run time: 52340 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190708 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f975aac0978f266e866546912c4d009f268593b, data reload: false

query1	980	372	388	372
query2	6540	2414	2342	2342
query3	6706	217	215	215
query4	33616	24321	23875	23875
query5	4343	624	462	462
query6	285	202	185	185
query7	4647	488	296	296
query8	309	259	239	239
query9	9517	2642	2632	2632
query10	456	312	251	251
query11	18290	15478	15275	15275
query12	167	112	108	108
query13	1693	529	424	424
query14	10820	7362	6898	6898
query15	265	197	193	193
query16	7505	602	453	453
query17	1577	747	580	580
query18	1991	418	296	296
query19	223	181	150	150
query20	119	112	113	112
query21	211	123	103	103
query22	4300	4293	4166	4166
query23	34212	33242	33632	33242
query24	6347	2251	2341	2251
query25	509	454	376	376
query26	916	274	168	168
query27	1985	445	334	334
query28	5212	2446	2397	2397
query29	695	534	441	441
query30	231	183	148	148
query31	964	913	827	827
query32	89	60	60	60
query33	494	354	295	295
query34	751	828	517	517
query35	809	812	740	740
query36	988	1006	954	954
query37	119	99	76	76
query38	4284	4138	4202	4138
query39	1513	1635	1444	1444
query40	206	114	102	102
query41	48	45	45	45
query42	118	103	103	103
query43	533	542	502	502
query44	1254	794	795	794
query45	180	181	171	171
query46	854	1029	641	641
query47	1935	1933	1880	1880
query48	373	391	328	328
query49	730	486	392	392
query50	617	638	393	393
query51	7003	6927	7082	6927
query52	110	102	92	92
query53	234	260	188	188
query54	479	483	405	405
query55	77	86	78	78
query56	248	257	261	257
query57	1242	1178	1147	1147
query58	240	227	228	227
query59	3090	3296	3019	3019
query60	301	251	257	251
query61	117	111	111	111
query62	887	806	754	754
query63	238	197	190	190
query64	4149	991	658	658
query65	3299	3193	3226	3193
query66	945	420	330	330
query67	15868	15746	15613	15613
query68	7742	693	516	516
query69	491	287	259	259
query70	1184	1143	1146	1143
query71	455	288	260	260
query72	6182	3881	3877	3877
query73	649	749	354	354
query74	10429	9120	8796	8796
query75	4023	3160	2645	2645
query76	3728	1181	773	773
query77	766	353	273	273
query78	10235	10261	9368	9368
query79	2819	821	600	600
query80	684	539	444	444
query81	497	282	232	232
query82	208	149	123	123
query83	173	164	145	145
query84	238	96	112	96
query85	780	358	313	313
query86	389	319	305	305
query87	4504	4739	4664	4664
query88	4484	2155	2130	2130
query89	394	338	304	304
query90	1937	182	187	182
query91	129	139	106	106
query92	69	64	53	53
query93	2293	866	542	542
query94	614	401	279	279
query95	334	268	257	257
query96	493	610	278	278
query97	2910	3005	2895	2895
query98	227	195	195	195
query99	1484	1559	1475	1475
Total cold run time: 291597 ms
Total hot run time: 190708 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f975aac0978f266e866546912c4d009f268593b, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.62	0.10	0.11
query5	0.42	0.42	0.41
query6	1.14	0.65	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.60	0.51	0.51
query10	0.55	0.56	0.55
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.61
query14	2.71	2.82	2.73
query15	0.88	0.82	0.81
query16	0.39	0.38	0.39
query17	0.97	1.06	0.99
query18	0.24	0.22	0.20
query19	1.90	1.75	2.01
query20	0.01	0.01	0.01
query21	15.37	0.91	0.57
query22	0.74	0.78	0.66
query23	15.27	1.41	0.64
query24	3.50	1.03	0.95
query25	0.21	0.11	0.07
query26	0.29	0.14	0.12
query27	0.09	0.08	0.04
query28	13.89	1.51	1.05
query29	12.58	3.89	3.26
query30	0.24	0.09	0.07
query31	2.81	0.58	0.39
query32	3.23	0.54	0.46
query33	3.13	3.13	3.14
query34	16.63	5.16	4.52
query35	4.58	4.48	4.54
query36	0.65	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.44 s
Total hot run time: 31.3 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.90% (10129/26039)
Line Coverage: 29.92% (85656/286306)
Region Coverage: 29.03% (43725/150621)
Branch Coverage: 25.57% (22316/87280)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3f975aac0978f266e866546912c4d009f268593b_3f975aac0978f266e866546912c4d009f268593b/report/index.html

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 2, 2025
Copy link
Contributor

github-actions bot commented Jan 2, 2025

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Jan 2, 2025

PR approved by anyone and no changes requested.

@BiteTheDDDDt BiteTheDDDDt merged commit 073fd68 into apache:master Jan 3, 2025
23 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants