Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](memory) max_sys_mem_available_low_water_mark_bytesdefault value modified to 6.4G #37174

Merged
merged 2 commits into from
Jul 5, 2024

Conversation

xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Jul 2, 2024

// The maximum low water mark of the system /proc/meminfo/MemAvailable, Unit byte, default 6.4G,
// actual low water mark=min(6.4G, MemTotal * 10%), avoid wasting too much memory on machines
// with large memory larger than 64G.
// Turn up max. On machines with more than 64G memory, more memory buffers will be reserved for Full GC.
// Turn down max. will use as much memory as possible.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Jul 2, 2024

run buildall

Copy link
Contributor

github-actions bot commented Jul 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39914 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 10669294d9df964acb4b634813db2fcf93e66935, data reload: false

------ Round 1 ----------------------------------
q1	18649	4636	4342	4342
q2	2024	189	188	188
q3	10565	1218	1015	1015
q4	10220	871	757	757
q5	7484	2659	2656	2656
q6	223	138	134	134
q7	945	593	595	593
q8	9225	2082	2096	2082
q9	8922	6519	6573	6519
q10	8979	3768	3755	3755
q11	455	230	239	230
q12	415	232	229	229
q13	17769	2967	3022	2967
q14	267	214	213	213
q15	529	483	493	483
q16	524	375	369	369
q17	979	687	695	687
q18	8209	7487	7378	7378
q19	6172	1478	1541	1478
q20	669	326	323	323
q21	4940	3202	3190	3190
q22	393	326	331	326
Total cold run time: 118557 ms
Total hot run time: 39914 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4421	4212	4231	4212
q2	362	258	267	258
q3	2972	2753	2715	2715
q4	1921	1610	1633	1610
q5	5268	5310	5309	5309
q6	216	129	135	129
q7	2100	1705	1769	1705
q8	3183	3383	3314	3314
q9	8345	8411	8419	8411
q10	3857	3664	3655	3655
q11	579	483	486	483
q12	794	631	606	606
q13	16373	3019	2979	2979
q14	283	250	268	250
q15	513	465	477	465
q16	464	418	418	418
q17	1829	1511	1472	1472
q18	7654	7439	7498	7439
q19	1710	1583	1489	1489
q20	1963	1818	1766	1766
q21	10643	4869	4615	4615
q22	609	529	543	529
Total cold run time: 76059 ms
Total hot run time: 53829 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172194 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 10669294d9df964acb4b634813db2fcf93e66935, data reload: false

query1	940	390	374	374
query2	6483	2389	2318	2318
query3	6700	213	216	213
query4	21192	17503	17138	17138
query5	4220	484	500	484
query6	265	178	157	157
query7	4609	305	285	285
query8	309	269	285	269
query9	8495	2366	2336	2336
query10	596	304	279	279
query11	10497	10071	10007	10007
query12	137	81	87	81
query13	1633	357	349	349
query14	9828	7684	6866	6866
query15	227	189	185	185
query16	7837	276	270	270
query17	1721	551	513	513
query18	1965	267	269	267
query19	184	151	148	148
query20	91	80	80	80
query21	211	126	130	126
query22	4192	4076	3970	3970
query23	33653	33219	33296	33219
query24	12057	2776	2838	2776
query25	687	363	365	363
query26	1827	154	148	148
query27	2987	312	310	310
query28	7191	2023	2040	2023
query29	1156	613	607	607
query30	268	150	153	150
query31	951	729	756	729
query32	95	53	53	53
query33	774	317	288	288
query34	949	457	460	457
query35	732	622	626	622
query36	1077	929	959	929
query37	273	78	76	76
query38	2916	2751	2771	2751
query39	848	799	816	799
query40	275	126	121	121
query41	55	50	53	50
query42	123	96	103	96
query43	590	534	526	526
query44	1226	709	713	709
query45	191	159	162	159
query46	1085	749	697	697
query47	1846	1761	1768	1761
query48	363	299	299	299
query49	1197	408	397	397
query50	763	373	381	373
query51	6968	6760	6794	6760
query52	105	88	95	88
query53	356	288	293	288
query54	971	438	429	429
query55	78	75	73	73
query56	284	259	271	259
query57	1195	1061	1043	1043
query58	262	257	256	256
query59	3450	3124	3186	3124
query60	312	288	276	276
query61	91	88	88	88
query62	654	448	453	448
query63	309	285	288	285
query64	9906	2272	1765	1765
query65	3176	3116	3097	3097
query66	1370	330	338	330
query67	15422	14832	15156	14832
query68	4680	536	537	536
query69	553	372	330	330
query70	1182	1097	1115	1097
query71	392	285	271	271
query72	7400	5178	5257	5178
query73	756	321	319	319
query74	5841	5502	5588	5502
query75	3491	2712	2646	2646
query76	2769	983	938	938
query77	566	305	299	299
query78	10156	9833	9786	9786
query79	1585	513	508	508
query80	1091	472	468	468
query81	536	221	221	221
query82	1058	103	106	103
query83	255	167	165	165
query84	234	85	87	85
query85	1267	286	268	268
query86	378	331	313	313
query87	3270	3086	3091	3086
query88	2871	2362	2333	2333
query89	471	384	376	376
query90	1813	191	191	191
query91	137	113	108	108
query92	63	50	54	50
query93	1371	496	496	496
query94	1231	198	271	198
query95	406	320	310	310
query96	576	265	265	265
query97	3191	3091	3067	3067
query98	228	199	207	199
query99	1284	861	838	838
Total cold run time: 274498 ms
Total hot run time: 172194 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 10669294d9df964acb4b634813db2fcf93e66935, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.04
query4	1.68	0.08	0.10
query5	0.50	0.49	0.49
query6	1.13	0.73	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.50
query10	0.55	0.55	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.13
query13	0.59	0.59	0.59
query14	0.81	0.80	0.79
query15	0.85	0.81	0.80
query16	0.36	0.36	0.36
query17	0.98	0.99	1.02
query18	0.23	0.25	0.25
query19	1.82	1.67	1.78
query20	0.01	0.01	0.01
query21	15.46	0.77	0.67
query22	4.13	7.19	2.02
query23	18.29	1.50	1.26
query24	2.06	0.25	0.22
query25	0.14	0.08	0.08
query26	0.26	0.17	0.18
query27	0.09	0.08	0.07
query28	13.27	1.02	1.00
query29	12.59	3.30	3.31
query30	0.26	0.06	0.06
query31	2.88	0.39	0.38
query32	3.27	0.47	0.46
query33	2.85	2.97	2.90
query34	17.27	4.44	4.45
query35	4.51	4.50	4.50
query36	0.65	0.46	0.47
query37	0.18	0.15	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.03
query40	0.16	0.14	0.16
query41	0.10	0.05	0.05
query42	0.05	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.48 s
Total hot run time: 30.73 s

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 2, 2024
Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by anyone and no changes requested.

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Jul 3, 2024

run buildall

Copy link
Contributor

github-actions bot commented Jul 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40265 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a3a23237989a252c7e81d666c2e0977747fd6228, data reload: false

------ Round 1 ----------------------------------
q1	17820	4578	4563	4563
q2	3018	199	195	195
q3	11096	1236	1074	1074
q4	10543	744	852	744
q5	7584	2713	2686	2686
q6	227	141	142	141
q7	982	614	641	614
q8	9292	2121	2096	2096
q9	9004	6498	6471	6471
q10	8998	3774	3699	3699
q11	450	242	236	236
q12	463	235	228	228
q13	17897	2960	3002	2960
q14	270	245	219	219
q15	531	499	489	489
q16	520	398	380	380
q17	972	702	676	676
q18	8045	7432	7454	7432
q19	5470	1464	1527	1464
q20	663	316	306	306
q21	4972	3854	3244	3244
q22	398	348	348	348
Total cold run time: 119215 ms
Total hot run time: 40265 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4436	4255	4307	4255
q2	373	269	264	264
q3	2979	2744	2677	2677
q4	1908	1633	1621	1621
q5	5253	5272	5316	5272
q6	225	133	137	133
q7	2120	1775	1690	1690
q8	3179	3356	3318	3318
q9	8338	8479	8343	8343
q10	3887	3613	3659	3613
q11	580	482	482	482
q12	796	589	579	579
q13	17530	2997	3006	2997
q14	294	274	261	261
q15	523	474	483	474
q16	478	410	409	409
q17	1785	1477	1458	1458
q18	7753	7499	7469	7469
q19	2511	1664	1506	1506
q20	1979	1798	1766	1766
q21	4866	4716	4647	4647
q22	620	536	569	536
Total cold run time: 72413 ms
Total hot run time: 53770 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172897 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a3a23237989a252c7e81d666c2e0977747fd6228, data reload: false

query1	913	381	388	381
query2	6459	2485	2378	2378
query3	6653	205	212	205
query4	20973	17559	17043	17043
query5	4208	505	500	500
query6	264	193	166	166
query7	4603	303	293	293
query8	310	307	305	305
query9	8511	2377	2368	2368
query10	617	309	304	304
query11	10499	9993	9997	9993
query12	136	93	87	87
query13	1643	390	375	375
query14	10045	7640	6639	6639
query15	233	191	194	191
query16	7762	275	281	275
query17	1893	594	545	545
query18	1864	290	289	289
query19	207	159	162	159
query20	97	89	91	89
query21	217	133	128	128
query22	4332	4089	3955	3955
query23	33836	33093	33176	33093
query24	12099	2866	2836	2836
query25	677	397	401	397
query26	1806	166	160	160
query27	3019	328	323	323
query28	7716	2076	2066	2066
query29	1137	645	640	640
query30	276	153	155	153
query31	952	740	764	740
query32	96	56	58	56
query33	811	324	309	309
query34	943	486	487	486
query35	764	668	632	632
query36	1137	983	968	968
query37	281	80	80	80
query38	2849	2728	2774	2728
query39	901	821	804	804
query40	285	132	134	132
query41	60	55	57	55
query42	127	104	105	104
query43	583	561	539	539
query44	1231	743	752	743
query45	197	168	170	168
query46	1104	754	750	750
query47	1843	1751	1781	1751
query48	380	311	308	308
query49	1218	440	439	439
query50	793	397	408	397
query51	6937	6806	6717	6717
query52	107	99	102	99
query53	391	304	303	303
query54	954	459	465	459
query55	75	74	78	74
query56	321	292	291	291
query57	1144	1038	1067	1038
query58	268	260	375	260
query59	3350	3036	3196	3036
query60	309	289	285	285
query61	92	90	96	90
query62	655	438	451	438
query63	342	302	302	302
query64	9818	2307	1798	1798
query65	3242	3164	3113	3113
query66	1340	351	329	329
query67	15636	14876	14818	14818
query68	8702	555	572	555
query69	759	423	370	370
query70	1252	1157	1134	1134
query71	527	289	285	285
query72	8918	5346	5356	5346
query73	1868	334	335	334
query74	6037	5459	5473	5459
query75	5308	2695	2670	2670
query76	5039	950	1007	950
query77	768	313	307	307
query78	10472	9940	9688	9688
query79	7391	529	530	529
query80	1008	503	483	483
query81	540	217	231	217
query82	384	116	112	112
query83	341	176	176	176
query84	265	88	88	88
query85	1032	285	268	268
query86	361	317	302	302
query87	3256	3155	3085	3085
query88	4912	2493	2460	2460
query89	523	400	400	400
query90	2124	193	197	193
query91	130	100	103	100
query92	68	50	49	49
query93	5622	529	527	527
query94	1407	191	195	191
query95	415	322	325	322
query96	612	279	273	273
query97	3200	3008	3045	3008
query98	220	206	200	200
query99	1133	862	841	841
Total cold run time: 299135 ms
Total hot run time: 172897 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a3a23237989a252c7e81d666c2e0977747fd6228, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.10	0.09
query5	0.51	0.48	0.47
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.53	0.53	0.54
query11	0.15	0.12	0.11
query12	0.16	0.12	0.13
query13	0.59	0.61	0.58
query14	0.79	0.78	0.79
query15	0.85	0.82	0.81
query16	0.36	0.36	0.37
query17	0.98	0.98	0.97
query18	0.23	0.23	0.26
query19	1.79	1.78	1.69
query20	0.01	0.01	0.00
query21	15.44	0.76	0.66
query22	4.20	7.41	1.66
query23	18.25	1.49	1.26
query24	2.19	0.24	0.22
query25	0.15	0.09	0.07
query26	0.27	0.17	0.18
query27	0.08	0.08	0.07
query28	13.20	1.02	1.00
query29	12.64	3.22	3.27
query30	0.26	0.07	0.06
query31	2.88	0.39	0.39
query32	3.25	0.47	0.46
query33	2.93	2.87	2.93
query34	17.25	4.43	4.44
query35	4.46	4.48	4.50
query36	0.65	0.46	0.48
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.20	0.15	0.14
query41	0.10	0.05	0.06
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.62 s
Total hot run time: 30.24 s

Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit 473c7ed into apache:master Jul 5, 2024
24 of 27 checks passed
dataroaring pushed a commit that referenced this pull request Jul 10, 2024
## Proposed changes

#37174 increased max sys_mem_available_low_water_mark from 1.6G to 6.4G,
causing memtable memory limiter reserving too much memory.

This PR adds a BE config `memtable_limiter_reserved_memory_bytes`,
defaults to 800MB, which defines bytes reserved in memtable memory
limiter.
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Jul 11, 2024
…ue modified to 6.4G (apache#37174)

// The maximum low water mark of the system
`/proc/meminfo/MemAvailable`, Unit byte, default 6.4G,
// actual low water mark=min(6.4G, MemTotal * 10%), avoid wasting too
much memory on machines
// with large memory larger than 64G.
// Turn up max. On machines with more than 64G memory, more memory
buffers will be reserved for Full GC.
// Turn down max. will use as much memory as possible.
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Jul 12, 2024
…ue modified to 6.4G (apache#37174)

// The maximum low water mark of the system
`/proc/meminfo/MemAvailable`, Unit byte, default 6.4G,
// actual low water mark=min(6.4G, MemTotal * 10%), avoid wasting too
much memory on machines
// with large memory larger than 64G.
// Turn up max. On machines with more than 64G memory, more memory
buffers will be reserved for Full GC.
// Turn down max. will use as much memory as possible.
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
)

## Proposed changes

apache#37174 increased max sys_mem_available_low_water_mark from 1.6G to 6.4G,
causing memtable memory limiter reserving too much memory.

This PR adds a BE config `memtable_limiter_reserved_memory_bytes`,
defaults to 800MB, which defines bytes reserved in memtable memory
limiter.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
…ue modified to 6.4G (#37174)

// The maximum low water mark of the system
`/proc/meminfo/MemAvailable`, Unit byte, default 6.4G,
// actual low water mark=min(6.4G, MemTotal * 10%), avoid wasting too
much memory on machines
// with large memory larger than 64G.
// Turn up max. On machines with more than 64G memory, more memory
buffers will be reserved for Full GC.
// Turn down max. will use as much memory as possible.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
## Proposed changes

#37174 increased max sys_mem_available_low_water_mark from 1.6G to 6.4G,
causing memtable memory limiter reserving too much memory.

This PR adds a BE config `memtable_limiter_reserved_memory_bytes`,
defaults to 800MB, which defines bytes reserved in memtable memory
limiter.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.x dev/2.1.5-merged dev/3.0.1-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants