Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[conf](parallel) Reduce parallel tasks for large cluster #38196

Merged
merged 1 commit into from
Jul 23, 2024
Merged

[conf](parallel) Reduce parallel tasks for large cluster #38196

merged 1 commit into from
Jul 23, 2024

Conversation

Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Jul 22, 2024 •
edited
Loading

Proposed changes

For large cluster, too many parallel tasks will cause performance issue. So this PR limit the max parallel tasks in Doris.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

@yiguolei yiguolei added dev/2.1.x usercase Important user case type label labels Jul 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39940 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 03c2d2d18544b8eb5cc45a564dd67829278c2c74, data reload: false

------ Round 1 ----------------------------------
q1	17623	4534	4259	4259
q2	2005	189	186	186
q3	10459	1250	1139	1139
q4	10187	778	915	778
q5	7617	2803	2628	2628
q6	219	136	137	136
q7	956	597	607	597
q8	9213	2090	2115	2090
q9	8756	6600	6589	6589
q10	8883	3782	3720	3720
q11	473	236	243	236
q12	418	229	229	229
q13	17780	2968	2986	2968
q14	286	226	242	226
q15	522	491	482	482
q16	519	389	377	377
q17	969	681	754	681
q18	8096	7449	7408	7408
q19	7802	1464	1446	1446
q20	696	333	325	325
q21	4847	3155	3172	3155
q22	348	293	285	285
Total cold run time: 118674 ms
Total hot run time: 39940 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4362	4298	4277	4277
q2	373	261	257	257
q3	3035	2912	3002	2912
q4	2009	1673	1726	1673
q5	5578	5527	5480	5480
q6	223	137	141	137
q7	2307	1852	1873	1852
q8	3262	3436	3447	3436
q9	8788	8880	8871	8871
q10	4197	3779	3890	3779
q11	597	490	505	490
q12	827	662	645	645
q13	17004	3110	3204	3110
q14	318	282	282	282
q15	548	494	490	490
q16	498	430	434	430
q17	1837	1528	1493	1493
q18	8107	8012	7879	7879
q19	1816	1511	1523	1511
q20	2130	1887	1885	1885
q21	10067	4876	4905	4876
q22	588	498	504	498
Total cold run time: 78471 ms
Total hot run time: 56263 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173913 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 03c2d2d18544b8eb5cc45a564dd67829278c2c74, data reload: false

query1	926	373	369	369
query2	6432	1966	1861	1861
query3	6638	210	224	210
query4	21173	17619	17568	17568
query5	3770	479	496	479
query6	256	188	180	180
query7	4594	292	286	286
query8	250	194	195	194
query9	8870	2454	2435	2435
query10	435	304	277	277
query11	11019	10243	10140	10140
query12	114	88	81	81
query13	1648	372	369	369
query14	10389	7512	7773	7512
query15	213	163	165	163
query16	7655	494	458	458
query17	1568	558	529	529
query18	1847	277	283	277
query19	202	150	161	150
query20	91	84	82	82
query21	197	127	126	126
query22	4405	4203	4219	4203
query23	34280	33454	33727	33454
query24	11068	2977	2935	2935
query25	656	410	399	399
query26	1116	157	161	157
query27	2496	273	280	273
query28	7092	2054	2048	2048
query29	910	631	657	631
query30	256	155	149	149
query31	955	771	746	746
query32	102	52	58	52
query33	733	334	335	334
query34	956	491	502	491
query35	885	761	754	754
query36	1148	992	956	956
query37	158	94	80	80
query38	2907	2860	2830	2830
query39	932	897	847	847
query40	195	118	119	118
query41	47	44	43	43
query42	113	98	100	98
query43	494	465	475	465
query44	1156	713	721	713
query45	190	164	165	164
query46	1099	755	732	732
query47	1884	1781	1821	1781
query48	379	295	285	285
query49	836	401	410	401
query50	775	387	390	387
query51	6823	6746	6709	6709
query52	99	93	88	88
query53	363	288	281	281
query54	895	445	439	439
query55	74	73	72	72
query56	293	264	269	264
query57	1165	1068	1025	1025
query58	256	246	258	246
query59	3050	2604	2689	2604
query60	293	280	267	267
query61	128	94	97	94
query62	809	648	648	648
query63	318	282	281	281
query64	9445	2248	1684	1684
query65	3162	3089	3144	3089
query66	704	323	324	323
query67	15516	14961	15137	14961
query68	4444	527	522	522
query69	464	332	342	332
query70	1213	1086	1064	1064
query71	407	280	265	265
query72	7062	5094	5794	5094
query73	752	322	320	320
query74	6166	5668	5649	5649
query75	3366	2698	2727	2698
query76	2725	967	959	959
query77	454	318	316	316
query78	10656	8982	9418	8982
query79	2587	514	509	509
query80	2641	518	467	467
query81	598	221	223	221
query82	794	134	135	134
query83	309	164	168	164
query84	273	85	87	85
query85	2192	322	296	296
query86	495	328	319	319
query87	3310	3139	3101	3101
query88	4115	2449	2472	2449
query89	484	383	389	383
query90	1924	197	193	193
query91	126	102	99	99
query92	62	52	51	51
query93	2447	497	486	486
query94	1292	294	311	294
query95	407	325	324	324
query96	639	281	273	273
query97	3175	3060	3060	3060
query98	216	199	197	197
query99	1547	1265	1260	1260
Total cold run time: 277245 ms
Total hot run time: 173913 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.69 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 03c2d2d18544b8eb5cc45a564dd67829278c2c74, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.68	0.10	0.09
query5	0.50	0.48	0.49
query6	1.12	0.73	0.73
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.54	0.50	0.48
query10	0.53	0.53	0.54
query11	0.16	0.11	0.11
query12	0.15	0.13	0.13
query13	0.60	0.59	0.59
query14	0.77	0.76	0.78
query15	0.87	0.82	0.81
query16	0.36	0.36	0.36
query17	1.03	0.98	1.03
query18	0.23	0.22	0.22
query19	1.88	1.67	1.84
query20	0.02	0.01	0.01
query21	15.39	0.74	0.65
query22	4.32	6.71	2.11
query23	18.28	1.44	1.20
query24	2.07	0.24	0.22
query25	0.15	0.08	0.08
query26	0.30	0.22	0.21
query27	0.45	0.23	0.22
query28	13.29	1.02	0.99
query29	12.64	3.38	3.36
query30	0.25	0.07	0.06
query31	2.87	0.40	0.38
query32	3.26	0.47	0.47
query33	2.90	2.93	2.86
query34	16.99	4.32	4.33
query35	4.46	4.44	4.39
query36	0.65	0.47	0.46
query37	0.19	0.16	0.17
query38	0.14	0.15	0.14
query39	0.04	0.04	0.03
query40	0.15	0.13	0.12
query41	0.09	0.04	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.84 s
Total hot run time: 30.69 s

@Gabriel39 Gabriel39 merged commit 7211c2d into apache:master Jul 23, 2024
29 of 31 checks passed
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 23, 2024
@Gabriel39
[conf](parallel) Reduce parallel tasks for large cluster (apache#38196)
6951009 
For large cluster, too many parallel tasks will cause performance issue.
So this PR limit the max parallel tasks in Doris.
@Gabriel39 Gabriel39 mentioned this pull request Jul 23, 2024
Merged
yiguolei pushed a commit that referenced this pull request Jul 23, 2024
@Gabriel39
[conf](parallel) Reduce parallel tasks for large cluster (#38196) (#3…
33e04b1 
…8233)

For large cluster, too many parallel tasks will cause performance issue.
So this PR limit the max parallel tasks in Doris.

pick #38196

<!--Describe your changes.-->
dataroaring pushed a commit that referenced this pull request Jul 24, 2024
@Gabriel39 @dataroaring
[conf](parallel) Reduce parallel tasks for large cluster (#38196)
fd2f06e 
For large cluster, too many parallel tasks will cause performance issue.
So this PR limit the max parallel tasks in Doris.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiguolei yiguolei yiguolei approved these changes

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged kind/behavior-changed reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Gabriel39 @doris-robot @BiteTheDDDDt @yiguolei @dataroaring