Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhancement](compaction) adjust compaction concurrency based on compaction score and workload #36672

Merged
merged 1 commit into from
Jun 26, 2024

Conversation

luwei16
Copy link
Contributor

@luwei16 luwei16 commented Jun 21, 2024

1 Resolved the issue where the priority queue did not reserve slots for cumulative compaction.

2 When considering compaction task priorities, introduced metrics for CPU and memory usage rates. When the compaction score is low, and CPU or memory usage is high, reduce the number of compaction tasks generated and allocate CPU and memory resources to queries or load.

3 Integrated the logic of the priority queue and concurrency control together, removing the previous priority code.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@luwei16
Copy link
Contributor Author

luwei16 commented Jun 21, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39990 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3bfa7c195b8431adfbbb7fe0bc268773a4e3bdb9, data reload: false

------ Round 1 ----------------------------------
q1	17604	4516	4254	4254
q2	2032	188	200	188
q3	10455	1130	1242	1130
q4	10202	830	790	790
q5	7477	2684	2655	2655
q6	225	136	134	134
q7	963	636	600	600
q8	9223	2111	2073	2073
q9	8756	6502	6489	6489
q10	8786	3730	3747	3730
q11	460	243	252	243
q12	412	238	233	233
q13	17758	3042	2964	2964
q14	270	226	223	223
q15	523	471	465	465
q16	511	374	377	374
q17	983	724	752	724
q18	8083	7571	7426	7426
q19	2903	1506	1476	1476
q20	651	318	333	318
q21	4859	3166	4065	3166
q22	399	345	335	335
Total cold run time: 113535 ms
Total hot run time: 39990 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4349	4230	4216	4216
q2	374	261	280	261
q3	2973	2784	2769	2769
q4	1897	1638	1607	1607
q5	5252	5286	5278	5278
q6	219	126	129	126
q7	2123	1707	1722	1707
q8	3210	3356	3324	3324
q9	8397	8338	8341	8338
q10	3878	3686	3691	3686
q11	568	506	496	496
q12	787	609	638	609
q13	17367	2963	2967	2963
q14	299	268	258	258
q15	514	473	469	469
q16	470	415	420	415
q17	1795	1469	1485	1469
q18	7741	7508	7436	7436
q19	1781	1551	1578	1551
q20	1959	1761	1757	1757
q21	4814	4798	4759	4759
q22	620	552	538	538
Total cold run time: 71387 ms
Total hot run time: 54032 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172246 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3bfa7c195b8431adfbbb7fe0bc268773a4e3bdb9, data reload: false

query1	919	393	374	374
query2	6454	2413	2368	2368
query3	6647	215	217	215
query4	19112	17117	17369	17117
query5	4172	485	482	482
query6	240	171	163	163
query7	4608	289	288	288
query8	308	271	282	271
query9	8538	2438	2415	2415
query10	623	309	288	288
query11	10561	9982	9980	9980
query12	139	93	84	84
query13	1640	377	365	365
query14	10232	6879	7663	6879
query15	232	194	181	181
query16	7878	264	262	262
query17	1909	543	519	519
query18	2003	272	282	272
query19	197	161	155	155
query20	88	82	82	82
query21	208	126	123	123
query22	4388	4036	3974	3974
query23	33671	33093	32975	32975
query24	11863	2880	2839	2839
query25	651	378	366	366
query26	1755	152	151	151
query27	3007	319	317	317
query28	7654	2082	2070	2070
query29	1063	628	606	606
query30	293	160	152	152
query31	973	732	753	732
query32	94	64	57	57
query33	779	292	266	266
query34	977	488	471	471
query35	732	625	611	611
query36	1102	970	972	970
query37	181	71	75	71
query38	2865	2742	2706	2706
query39	886	812	796	796
query40	274	134	128	128
query41	55	54	51	51
query42	123	104	101	101
query43	587	542	550	542
query44	1244	736	732	732
query45	200	169	165	165
query46	1081	741	726	726
query47	1879	1808	1771	1771
query48	365	295	286	286
query49	1182	412	404	404
query50	763	378	389	378
query51	6847	6759	6681	6681
query52	102	96	89	89
query53	370	297	284	284
query54	914	455	437	437
query55	77	73	76	73
query56	285	257	267	257
query57	1180	1051	1036	1036
query58	259	236	268	236
query59	3325	3137	2991	2991
query60	294	277	279	277
query61	95	93	95	93
query62	647	442	435	435
query63	309	293	290	290
query64	9820	2262	1751	1751
query65	3216	3083	3103	3083
query66	1374	327	332	327
query67	15344	14822	15013	14822
query68	4539	540	539	539
query69	445	312	321	312
query70	1123	1083	1132	1083
query71	396	281	278	278
query72	7036	5501	5849	5501
query73	741	325	321	321
query74	6038	5810	5542	5542
query75	3603	2682	2692	2682
query76	2795	935	913	913
query77	448	308	295	295
query78	10203	9874	9666	9666
query79	2089	523	515	515
query80	801	485	465	465
query81	578	221	219	219
query82	1076	105	104	104
query83	264	178	172	172
query84	244	87	145	87
query85	1560	281	278	278
query86	482	317	326	317
query87	3246	3102	3091	3091
query88	4025	2353	2347	2347
query89	466	377	414	377
query90	1761	198	192	192
query91	129	100	102	100
query92	65	50	53	50
query93	2301	511	505	505
query94	1229	184	189	184
query95	406	306	322	306
query96	597	264	268	264
query97	3282	3072	3058	3058
query98	215	195	199	195
query99	1215	846	829	829
Total cold run time: 275251 ms
Total hot run time: 172246 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.47% (9008/24699)
Line Coverage: 28.02% (73894/263705)
Region Coverage: 27.50% (38383/139578)
Branch Coverage: 24.19% (19558/80852)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3bfa7c195b8431adfbbb7fe0bc268773a4e3bdb9_3bfa7c195b8431adfbbb7fe0bc268773a4e3bdb9/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 31.29 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3bfa7c195b8431adfbbb7fe0bc268773a4e3bdb9, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.66	0.08	0.08
query5	0.51	0.49	0.50
query6	1.14	0.74	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.55	0.48	0.49
query10	0.54	0.56	0.55
query11	0.15	0.12	0.12
query12	0.14	0.13	0.13
query13	0.60	0.58	0.57
query14	0.76	0.76	0.79
query15	0.85	0.82	0.82
query16	0.38	0.35	0.37
query17	0.98	0.95	0.98
query18	0.23	0.22	0.26
query19	1.76	1.71	1.77
query20	0.02	0.01	0.01
query21	15.45	0.67	0.66
query22	3.83	6.89	2.68
query23	18.27	1.40	1.25
query24	2.07	0.25	0.22
query25	0.15	0.08	0.09
query26	0.27	0.18	0.18
query27	0.07	0.07	0.08
query28	13.24	1.02	0.99
query29	12.66	3.28	3.24
query30	0.26	0.07	0.06
query31	2.85	0.39	0.39
query32	3.29	0.48	0.47
query33	2.90	2.90	2.96
query34	17.26	4.41	4.37
query35	4.49	4.50	4.50
query36	0.65	0.46	0.47
query37	0.19	0.17	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.17	0.15	0.15
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.03	0.04
Total cold run time: 109.16 s
Total hot run time: 31.29 s

@luwei16 luwei16 force-pushed the luwei/fix-compaction-policy branch 2 times, most recently from 3ac366e to 3bfa7c1 Compare June 24, 2024 16:25
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 26, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 2c3a96e into apache:master Jun 26, 2024
41 of 46 checks passed
dataroaring pushed a commit that referenced this pull request Jun 28, 2024
…action score and workload (#36672)

1 Resolved the issue where the priority queue did not reserve slots for
cumulative compaction.

2 When considering compaction task priorities, introduced metrics for
CPU and memory usage rates. When the compaction score is low, and CPU or
memory usage is high, reduce the number of compaction tasks generated
and allocate CPU and memory resources to queries or load.

3 Integrated the logic of the priority queue and concurrency control
together, removing the previous priority code.
luwei16 added a commit to luwei16/incubator-doris that referenced this pull request Jul 8, 2024
…action score and workload (apache#36672)

1 Resolved the issue where the priority queue did not reserve slots for
cumulative compaction.

2 When considering compaction task priorities, introduced metrics for
CPU and memory usage rates. When the compaction score is low, and CPU or
memory usage is high, reduce the number of compaction tasks generated
and allocate CPU and memory resources to queries or load.

3 Integrated the logic of the priority queue and concurrency control
together, removing the previous priority code.
dataroaring pushed a commit that referenced this pull request Jul 9, 2024
…paction score and workload (#37491)

adjust compaction concurrency based on compaction score and workload
#36672
fix null pointer when retrieving CPU load average #37171
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants