Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](memory) Optimize ClearCacheActionimplementation #38438

Merged
merged 4 commits into from
Aug 2, 2024
Merged

[opt](memory) Optimize ClearCacheActionimplementation #38438

merged 4 commits into from
Aug 2, 2024

Conversation

xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Jul 28, 2024 •
edited
Loading

  1. ClearCacheAction support for specifying cache, http://beip:8040/api/clear_cache/all, http://beip:8040/api/clear_cache/DataPageCache
  2. Optimize result returned by ClearCacheAction.
  3. Optimize Allocator large memory printing stacktrace
  4. cache_prune_stale_thread no longer check conf to prune cache, which has delays and brings a bad experience.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39816 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7b9b20b63343b247e6cd88e87bbdb7b7d1799518, data reload: false

------ Round 1 ----------------------------------
q1	17619	4564	4334	4334
q2	2018	201	204	201
q3	10585	1363	1152	1152
q4	10614	766	706	706
q5	7989	2813	2728	2728
q6	226	147	143	143
q7	1012	634	621	621
q8	9516	1986	1971	1971
q9	9170	6696	6733	6696
q10	9338	3851	3769	3769
q11	457	251	244	244
q12	401	227	221	221
q13	17773	2963	2974	2963
q14	288	250	241	241
q15	509	471	500	471
q16	495	381	382	381
q17	967	685	652	652
q18	8118	7534	7495	7495
q19	6251	962	976	962
q20	695	338	342	338
q21	5015	3243	3279	3243
q22	354	284	289	284
Total cold run time: 119410 ms
Total hot run time: 39816 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4544	4298	4272	4272
q2	379	275	272	272
q3	3093	2922	2932	2922
q4	2015	1752	1687	1687
q5	5541	5610	5507	5507
q6	238	142	139	139
q7	2281	1882	1941	1882
q8	3292	3473	3431	3431
q9	8737	8711	8821	8711
q10	4108	3887	3876	3876
q11	596	507	534	507
q12	804	633	638	633
q13	16824	3256	3172	3172
q14	341	288	283	283
q15	540	485	492	485
q16	512	447	435	435
q17	1831	1520	1548	1520
q18	8342	7985	7705	7705
q19	1773	1484	1720	1484
q20	2178	1875	1864	1864
q21	5198	5003	5021	5003
q22	620	538	536	536
Total cold run time: 73787 ms
Total hot run time: 56326 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173290 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7b9b20b63343b247e6cd88e87bbdb7b7d1799518, data reload: false

query1	914	376	375	375
query2	6304	1977	1867	1867
query3	6657	206	218	206
query4	28287	17441	17335	17335
query5	3630	483	491	483
query6	263	170	173	170
query7	4580	293	291	291
query8	247	203	205	203
query9	8555	2494	2470	2470
query10	442	283	275	275
query11	11565	10014	10054	10014
query12	138	87	86	86
query13	1633	374	370	370
query14	10315	7753	7891	7753
query15	225	173	167	167
query16	7678	489	469	469
query17	1547	582	548	548
query18	1979	290	316	290
query19	197	151	152	151
query20	94	89	86	86
query21	208	108	107	107
query22	4349	4105	3948	3948
query23	34197	33805	33713	33713
query24	12038	2975	2981	2975
query25	704	421	417	417
query26	1761	162	160	160
query27	3022	282	290	282
query28	7409	2093	2080	2080
query29	1182	471	457	457
query30	256	156	165	156
query31	1016	774	758	758
query32	104	58	58	58
query33	790	367	359	359
query34	877	488	515	488
query35	896	763	774	763
query36	1119	961	960	960
query37	288	88	90	88
query38	3010	2965	2827	2827
query39	927	830	829	829
query40	244	114	115	114
query41	47	45	42	42
query42	110	92	98	92
query43	496	479	476	476
query44	1202	731	725	725
query45	210	172	173	172
query46	1099	749	712	712
query47	1830	1752	1782	1752
query48	367	291	291	291
query49	931	409	410	409
query50	793	400	409	400
query51	6801	6636	6680	6636
query52	99	91	103	91
query53	255	181	187	181
query54	958	454	440	440
query55	76	76	73	73
query56	287	272	275	272
query57	1149	1085	1043	1043
query58	264	301	292	292
query59	2968	2791	2671	2671
query60	302	285	300	285
query61	97	93	93	93
query62	805	645	653	645
query63	218	186	183	183
query64	10315	2294	1707	1707
query65	3182	3098	3113	3098
query66	1246	355	342	342
query67	15597	14694	14573	14573
query68	8718	543	541	541
query69	721	387	345	345
query70	1361	1116	1048	1048
query71	542	274	269	269
query72	8543	5901	5334	5334
query73	2277	334	326	326
query74	6146	5660	5576	5576
query75	5144	2682	2650	2650
query76	5523	924	919	919
query77	769	323	306	306
query78	9725	9015	8951	8951
query79	8072	521	527	521
query80	879	483	494	483
query81	578	215	218	215
query82	284	132	130	130
query83	333	173	175	173
query84	265	77	78	77
query85	971	376	297	297
query86	351	310	296	296
query87	3325	3077	3060	3060
query88	4494	2491	2480	2480
query89	497	287	281	281
query90	2038	199	190	190
query91	129	101	100	100
query92	65	47	55	47
query93	5856	531	520	520
query94	1013	305	304	304
query95	360	256	258	256
query96	625	281	282	281
query97	3214	3014	3038	3014
query98	218	200	198	198
query99	1605	1258	1272	1258
Total cold run time: 305992 ms
Total hot run time: 173290 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.02 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7b9b20b63343b247e6cd88e87bbdb7b7d1799518, data reload: false

query1	0.05	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.05
query4	1.68	0.10	0.10
query5	0.50	0.48	0.48
query6	1.14	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.54	0.55	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.58	0.58
query14	0.78	0.77	0.79
query15	0.85	0.81	0.81
query16	0.37	0.37	0.38
query17	1.05	1.04	1.03
query18	0.22	0.22	0.22
query19	1.92	1.80	1.76
query20	0.01	0.01	0.01
query21	15.40	0.76	0.65
query22	3.76	6.68	2.17
query23	18.30	1.45	1.31
query24	2.08	0.23	0.22
query25	0.15	0.08	0.08
query26	0.28	0.21	0.21
query27	0.45	0.23	0.24
query28	13.28	1.02	1.00
query29	12.68	3.36	3.33
query30	0.25	0.06	0.06
query31	2.88	0.39	0.39
query32	3.29	0.48	0.46
query33	2.87	2.96	2.90
query34	17.03	4.32	4.34
query35	4.38	4.38	4.44
query36	0.65	0.47	0.47
query37	0.18	0.16	0.15
query38	0.17	0.15	0.14
query39	0.05	0.03	0.03
query40	0.16	0.12	0.12
query41	0.10	0.04	0.05
query42	0.06	0.05	0.06
query43	0.04	0.04	0.04
Total cold run time: 109.44 s
Total hot run time: 31.02 s

yiguolei
yiguolei reviewed Jul 29, 2024
View reviewed changes
be/src/common/config.cpp Outdated
@@ -133,13 +133,11 @@ DEFINE_mString(process_full_gc_size, "20%");
// If false, cancel query when the memory used exceeds exec_mem_limit, same as before.
DEFINE_mBool(enable_query_memory_overcommit, "true");

DEFINE_mBool(disable_memory_gc, "false");
DEFINE_mBool(enable_memory_reclamation, "true");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这些配置文件的名字先别改了，影响不大。另外，改了还得改文档，各个版本之间兼容还有问题。

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@xinyiZzz xinyiZzz force-pushed the 20240726_fix_memory branch 2 times, most recently from 87bebf2 to 0220155 Compare July 30, 2024 03:49
@xinyiZzz xinyiZzz force-pushed the 20240726_fix_memory branch from 0220155 to 2e5ac71 Compare July 30, 2024 03:50
@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 42065 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2e5ac718bd6846df0493c9386e51c5827e91cbed, data reload: false

------ Round 1 ----------------------------------
q1	18821	4267	4154	4154
q2	3019	209	214	209
q3	11639	1330	1364	1330
q4	10599	905	911	905
q5	8034	3009	2988	2988
q6	217	136	138	136
q7	1048	637	610	610
q8	9445	1950	1981	1950
q9	8536	6616	6623	6616
q10	8703	3871	3867	3867
q11	425	251	259	251
q12	416	232	228	228
q13	17757	2973	2936	2936
q14	276	248	243	243
q15	538	494	481	481
q16	494	401	400	400
q17	998	948	925	925
q18	8123	7268	7214	7214
q19	1377	1228	1235	1228
q20	575	336	339	336
q21	5408	4767	4819	4767
q22	360	291	291	291
Total cold run time: 116808 ms
Total hot run time: 42065 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4080	4042	4052	4042
q2	334	225	221	221
q3	2992	3009	2985	2985
q4	1901	1877	1881	1877
q5	5248	5253	5244	5244
q6	218	128	134	128
q7	2055	1701	1688	1688
q8	3226	3295	3315	3295
q9	8373	8325	8312	8312
q10	3785	3854	3826	3826
q11	547	464	451	451
q12	747	522	560	522
q13	12389	2935	2973	2935
q14	284	264	261	261
q15	511	480	476	476
q16	473	395	395	395
q17	1750	1704	1673	1673
q18	7754	7298	7229	7229
q19	1692	1701	1676	1676
q20	1960	1780	1755	1755
q21	5550	5227	5307	5227
q22	541	484	489	484
Total cold run time: 66410 ms
Total hot run time: 54702 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169380 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2e5ac718bd6846df0493c9386e51c5827e91cbed, data reload: false

query1	919	385	370	370
query2	6467	1670	1679	1670
query3	6667	210	223	210
query4	19457	17609	17338	17338
query5	4292	508	509	508
query6	273	172	179	172
query7	4608	302	298	298
query8	251	194	199	194
query9	8519	2364	2360	2360
query10	433	308	270	270
query11	10524	10011	9984	9984
query12	138	91	87	87
query13	1641	379	380	379
query14	10210	8488	7667	7667
query15	198	156	171	156
query16	7084	445	425	425
query17	954	541	537	537
query18	1921	282	301	282
query19	195	144	140	140
query20	89	85	86	85
query21	208	99	100	99
query22	4322	4067	3930	3930
query23	33751	32922	33248	32922
query24	10429	2990	3048	2990
query25	688	371	407	371
query26	1748	146	152	146
query27	2941	272	286	272
query28	6939	1986	1975	1975
query29	1312	429	420	420
query30	285	150	148	148
query31	948	740	734	734
query32	99	54	58	54
query33	708	314	313	313
query34	917	469	480	469
query35	860	733	718	718
query36	976	852	851	851
query37	289	78	78	78
query38	2834	2781	2751	2751
query39	849	833	805	805
query40	286	122	114	114
query41	48	48	45	45
query42	126	97	101	97
query43	469	430	410	410
query44	1221	733	729	729
query45	204	180	179	179
query46	1085	811	771	771
query47	1792	1709	1683	1683
query48	361	295	298	295
query49	1186	430	413	413
query50	890	428	433	428
query51	6813	6618	6650	6618
query52	103	96	94	94
query53	253	193	180	180
query54	650	461	457	457
query55	77	76	75	75
query56	277	254	268	254
query57	1189	1043	1042	1042
query58	272	263	269	263
query59	2532	2301	2305	2301
query60	303	277	289	277
query61	101	155	123	123
query62	935	647	663	647
query63	214	183	187	183
query64	5927	1918	1909	1909
query65	3129	3117	3107	3107
query66	1448	343	358	343
query67	15468	14814	14833	14814
query68	4312	613	597	597
query69	442	298	305	298
query70	1123	1056	1064	1056
query71	385	284	276	276
query72	7177	2692	2559	2559
query73	785	328	332	328
query74	6043	5592	5587	5587
query75	3389	2767	2675	2675
query76	2319	1333	1370	1333
query77	489	338	343	338
query78	9453	8883	8853	8853
query79	1993	529	541	529
query80	1185	504	501	501
query81	544	227	229	227
query82	1185	130	132	130
query83	242	180	169	169
query84	276	77	77	77
query85	1344	385	372	372
query86	392	290	311	290
query87	3333	3088	3081	3081
query88	2985	2426	2434	2426
query89	392	300	295	295
query90	1812	201	200	200
query91	141	114	112	112
query92	64	53	53	53
query93	1461	631	629	629
query94	895	315	316	315
query95	397	287	290	287
query96	598	282	282	282
query97	3173	3013	3037	3013
query98	216	205	200	200
query99	1637	1320	1263	1263
Total cold run time: 263484 ms
Total hot run time: 169380 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.89 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2e5ac718bd6846df0493c9386e51c5827e91cbed, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.69	0.07	0.07
query5	0.48	0.47	0.49
query6	1.14	0.70	0.71
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.59	0.54	0.51
query10	0.55	0.56	0.57
query11	0.15	0.12	0.12
query12	0.15	0.13	0.13
query13	0.61	0.59	0.59
query14	0.78	0.79	0.78
query15	0.90	0.85	0.86
query16	0.35	0.36	0.35
query17	1.00	1.00	0.97
query18	0.23	0.22	0.21
query19	1.84	1.72	1.73
query20	0.02	0.01	0.01
query21	15.40	0.78	0.66
query22	4.03	7.77	1.24
query23	17.84	1.37	1.35
query24	2.29	0.23	0.21
query25	0.19	0.09	0.07
query26	0.32	0.21	0.20
query27	0.46	0.25	0.23
query28	13.16	1.00	0.97
query29	12.55	3.35	3.26
query30	0.26	0.05	0.05
query31	2.86	0.40	0.40
query32	3.24	0.49	0.50
query33	2.95	2.91	2.96
query34	15.43	4.28	4.24
query35	4.29	4.32	4.31
query36	0.69	0.49	0.49
query37	0.19	0.18	0.16
query38	0.17	0.15	0.17
query39	0.04	0.03	0.04
query40	0.16	0.13	0.13
query41	0.10	0.04	0.05
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 107.61 s
Total hot run time: 29.89 s

@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41835 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66b115cf0316969c0634cceb03604e09ddfeba51, data reload: false

------ Round 1 ----------------------------------
q1	17640	4178	4070	4070
q2	2030	203	211	203
q3	10453	1454	1293	1293
q4	10157	886	913	886
q5	7632	3025	2996	2996
q6	222	137	135	135
q7	1047	611	612	611
q8	9449	1971	1945	1945
q9	8526	6675	6631	6631
q10	8788	3874	3867	3867
q11	432	250	254	250
q12	408	230	228	228
q13	17766	2946	2928	2928
q14	276	241	247	241
q15	517	488	475	475
q16	521	399	382	382
q17	984	945	910	910
q18	8054	7359	7299	7299
q19	1501	1224	1223	1223
q20	571	319	324	319
q21	5312	4658	4699	4658
q22	364	287	285	285
Total cold run time: 112650 ms
Total hot run time: 41835 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4099	4046	4058	4046
q2	326	224	227	224
q3	3003	2993	3179	2993
q4	2037	2026	1986	1986
q5	5608	5532	5465	5465
q6	234	131	127	127
q7	2135	1785	1806	1785
q8	3349	3389	3381	3381
q9	8809	8762	8749	8749
q10	3998	4026	4048	4026
q11	542	443	451	443
q12	740	579	577	577
q13	16272	3141	3088	3088
q14	318	256	275	256
q15	535	504	511	504
q16	473	424	420	420
q17	1759	1753	1778	1753
q18	8201	7803	7741	7741
q19	1753	1729	1717	1717
q20	2058	1850	1828	1828
q21	5839	5477	5498	5477
q22	532	450	486	450
Total cold run time: 72620 ms
Total hot run time: 57036 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170311 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 66b115cf0316969c0634cceb03604e09ddfeba51, data reload: false

query1	918	381	371	371
query2	6442	1698	1673	1673
query3	6640	214	226	214
query4	20232	17705	17359	17359
query5	3664	510	517	510
query6	267	163	176	163
query7	4588	300	291	291
query8	249	199	195	195
query9	8516	2368	2374	2368
query10	440	285	262	262
query11	10495	10084	10112	10084
query12	120	90	94	90
query13	1630	377	382	377
query14	9329	7743	7042	7042
query15	196	159	165	159
query16	6799	472	449	449
query17	930	581	534	534
query18	1882	282	273	273
query19	192	148	142	142
query20	88	86	85	85
query21	200	98	97	97
query22	4366	4073	3984	3984
query23	34003	33781	33505	33505
query24	9401	3096	3076	3076
query25	650	409	419	409
query26	1411	156	152	152
query27	2999	290	280	280
query28	7477	2059	2041	2041
query29	1028	426	438	426
query30	241	159	156	156
query31	936	760	750	750
query32	104	53	57	53
query33	678	317	329	317
query34	933	499	506	499
query35	860	764	766	764
query36	1111	892	887	887
query37	181	86	84	84
query38	2968	2886	2745	2745
query39	887	840	809	809
query40	254	117	115	115
query41	48	48	46	46
query42	125	98	95	95
query43	489	430	439	430
query44	1161	727	723	723
query45	210	185	186	185
query46	1109	831	818	818
query47	1836	1752	1707	1707
query48	370	300	292	292
query49	871	445	432	432
query50	922	427	431	427
query51	6792	6818	6701	6701
query52	106	87	96	87
query53	259	190	185	185
query54	605	466	468	466
query55	80	77	77	77
query56	299	258	259	258
query57	1112	1041	1074	1041
query58	282	279	287	279
query59	2746	2544	2553	2544
query60	314	291	293	291
query61	122	117	119	117
query62	888	672	673	672
query63	219	186	183	183
query64	4993	1937	1896	1896
query65	3164	3096	3094	3094
query66	985	335	325	325
query67	15460	15056	14852	14852
query68	4428	575	586	575
query69	477	297	351	297
query70	1098	1072	1089	1072
query71	440	270	279	270
query72	7840	2758	2549	2549
query73	770	336	332	332
query74	6041	5624	5652	5624
query75	3497	2722	2741	2722
query76	2576	1305	1418	1305
query77	489	308	310	308
query78	9541	9005	8918	8918
query79	2735	542	535	535
query80	1062	511	495	495
query81	570	224	230	224
query82	959	132	125	125
query83	266	166	163	163
query84	253	84	79	79
query85	1315	331	371	331
query86	427	313	293	293
query87	3227	3120	3104	3104
query88	3891	2387	2386	2386
query89	395	291	289	289
query90	1820	191	186	186
query91	127	101	100	100
query92	59	51	52	51
query93	2036	632	622	622
query94	868	301	297	297
query95	385	270	267	267
query96	603	278	283	278
query97	3202	3051	3001	3001
query98	234	200	204	200
query99	1619	1300	1304	1300
Total cold run time: 263122 ms
Total hot run time: 170311 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.3 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 66b115cf0316969c0634cceb03604e09ddfeba51, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.22	0.04	0.04
query4	1.68	0.07	0.07
query5	0.49	0.48	0.50
query6	1.13	0.72	0.71
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.58	0.51	0.51
query10	0.56	0.56	0.57
query11	0.16	0.11	0.11
query12	0.15	0.13	0.12
query13	0.61	0.59	0.61
query14	0.77	0.79	0.81
query15	0.91	0.88	0.88
query16	0.36	0.36	0.36
query17	0.98	0.99	0.99
query18	0.23	0.22	0.22
query19	1.86	1.81	1.83
query20	0.01	0.01	0.01
query21	15.47	0.78	0.66
query22	4.17	8.41	1.50
query23	18.06	1.32	1.33
query24	2.24	0.22	0.22
query25	0.18	0.08	0.08
query26	0.33	0.21	0.22
query27	0.45	0.24	0.23
query28	13.16	1.00	0.96
query29	12.62	3.34	3.29
query30	0.26	0.06	0.05
query31	2.88	0.40	0.40
query32	3.23	0.50	0.47
query33	2.94	2.96	2.94
query34	15.46	4.32	4.23
query35	4.33	4.32	4.32
query36	0.66	0.48	0.48
query37	0.18	0.16	0.15
query38	0.17	0.16	0.14
query39	0.04	0.03	0.04
query40	0.16	0.13	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 108.06 s
Total hot run time: 30.3 s

@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41408 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dc707718fef6740eaf41ec74d0a72f2e2513fbc2, data reload: false

------ Round 1 ----------------------------------
q1	18614	4249	4079	4079
q2	2021	214	216	214
q3	10424	1252	1323	1252
q4	10169	823	930	823
q5	7653	2889	2971	2889
q6	217	134	135	134
q7	1034	620	626	620
q8	9434	1844	1911	1844
q9	8529	6597	6580	6580
q10	8710	3833	3862	3833
q11	430	255	261	255
q12	415	227	227	227
q13	17757	2951	2948	2948
q14	270	251	247	247
q15	523	482	478	478
q16	486	396	383	383
q17	943	912	861	861
q18	8054	7328	7217	7217
q19	1388	1217	1213	1213
q20	572	315	352	315
q21	5322	4837	4718	4718
q22	353	283	278	278
Total cold run time: 113318 ms
Total hot run time: 41408 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4017	4067	4238	4067
q2	324	219	216	216
q3	2979	2984	2979	2979
q4	1867	1896	1841	1841
q5	5231	5228	5228	5228
q6	214	129	127	127
q7	2062	1628	1685	1628
q8	3198	3261	3261	3261
q9	8308	8257	8296	8257
q10	3768	3819	3829	3819
q11	550	436	454	436
q12	710	579	560	560
q13	16577	2934	2905	2905
q14	298	249	249	249
q15	519	479	476	476
q16	446	384	397	384
q17	1738	1690	1650	1650
q18	7769	7322	7230	7230
q19	1667	1672	1663	1663
q20	1977	1759	1766	1759
q21	5388	5205	5257	5205
q22	506	447	464	447
Total cold run time: 70113 ms
Total hot run time: 54387 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169187 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dc707718fef6740eaf41ec74d0a72f2e2513fbc2, data reload: false

query1	911	374	377	374
query2	6472	1730	1674	1674
query3	6688	208	223	208
query4	19096	17639	17507	17507
query5	4326	517	522	517
query6	291	173	178	173
query7	4602	291	284	284
query8	250	204	197	197
query9	8504	2356	2338	2338
query10	445	273	265	265
query11	10627	10138	10004	10004
query12	130	85	88	85
query13	1610	388	371	371
query14	9445	7166	7695	7166
query15	206	164	161	161
query16	7131	480	469	469
query17	979	566	565	565
query18	1919	294	292	292
query19	202	151	147	147
query20	95	89	87	87
query21	210	102	99	99
query22	4155	4185	3947	3947
query23	33666	32954	33086	32954
query24	10336	3145	3045	3045
query25	730	404	417	404
query26	1872	152	153	152
query27	3005	285	283	283
query28	7058	1976	1972	1972
query29	1394	432	429	429
query30	286	153	150	150
query31	969	819	740	740
query32	99	53	54	53
query33	700	302	307	302
query34	903	470	485	470
query35	852	718	728	718
query36	988	864	888	864
query37	304	77	76	76
query38	2865	2764	2772	2764
query39	869	815	821	815
query40	287	115	112	112
query41	48	45	44	44
query42	134	97	98	97
query43	479	413	458	413
query44	1195	723	722	722
query45	203	176	176	176
query46	1089	810	782	782
query47	1810	1738	1719	1719
query48	365	288	287	287
query49	1190	460	423	423
query50	893	436	437	436
query51	6803	6755	6677	6677
query52	102	86	94	86
query53	256	179	177	177
query54	633	472	460	460
query55	75	74	78	74
query56	295	265	272	265
query57	1137	1044	1023	1023
query58	283	261	263	261
query59	2835	2381	2449	2381
query60	312	278	277	277
query61	104	97	100	97
query62	916	651	658	651
query63	216	189	182	182
query64	5920	1927	1898	1898
query65	3135	3101	3070	3070
query66	1451	341	341	341
query67	15522	15205	14771	14771
query68	4409	586	601	586
query69	455	318	297	297
query70	1091	1056	1096	1056
query71	409	274	270	270
query72	7192	2690	2557	2557
query73	785	330	330	330
query74	6079	5641	5671	5641
query75	3406	2771	2743	2743
query76	2547	1308	1393	1308
query77	440	304	305	304
query78	9375	8925	8770	8770
query79	2716	527	534	527
query80	1214	511	508	508
query81	556	224	220	220
query82	789	132	126	126
query83	256	171	168	168
query84	265	81	87	81
query85	1721	368	297	297
query86	493	297	290	290
query87	3290	3106	3115	3106
query88	3839	2397	2381	2381
query89	381	300	296	296
query90	1878	203	186	186
query91	124	103	102	102
query92	59	50	49	49
query93	2353	629	632	629
query94	924	289	306	289
query95	384	276	267	267
query96	612	281	276	276
query97	3189	3037	3024	3024
query98	226	195	194	194
query99	1596	1286	1292	1286
Total cold run time: 265931 ms
Total hot run time: 169187 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.05 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dc707718fef6740eaf41ec74d0a72f2e2513fbc2, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.05	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.49	0.50
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.57	0.52	0.50
query10	0.54	0.56	0.56
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.61	0.61	0.60
query14	0.77	0.77	0.80
query15	0.89	0.86	0.85
query16	0.35	0.36	0.35
query17	0.97	0.99	1.03
query18	0.23	0.22	0.21
query19	1.82	1.74	1.77
query20	0.01	0.00	0.01
query21	15.49	0.77	0.66
query22	4.26	7.21	1.38
query23	17.99	1.31	1.25
query24	2.24	0.22	0.22
query25	0.20	0.08	0.08
query26	0.31	0.22	0.21
query27	0.47	0.23	0.24
query28	13.17	1.00	0.97
query29	12.56	3.30	3.31
query30	0.26	0.06	0.05
query31	2.86	0.40	0.41
query32	3.25	0.49	0.48
query33	2.91	2.97	2.98
query34	15.44	4.29	4.26
query35	4.28	4.27	4.30
query36	0.68	0.48	0.47
query37	0.18	0.17	0.16
query38	0.16	0.16	0.15
query39	0.05	0.03	0.03
query40	0.15	0.14	0.15
query41	0.10	0.05	0.04
query42	0.05	0.04	0.04
query43	0.04	0.05	0.04
Total cold run time: 107.91 s
Total hot run time: 30.05 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 1, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

PR approved by anyone and no changes requested.

wangbo
wangbo approved these changes Aug 1, 2024
View reviewed changes
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit 55d63a8 into apache:master Aug 2, 2024
27 of 30 checks passed
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@xinyiZzz @dataroaring
[opt](memory) Optimize ClearCacheActionimplementation (#38438)
4ebfa3f 
1. `ClearCacheAction` support for specifying cache,
`http://beip:8040/api/clear_cache/all`,
`http://beip:8040/api/clear_cache/DataPageCache`
2. Optimize result returned by `ClearCacheAction`.
3. Optimize Allocator large memory printing stacktrace
4. `cache_prune_stale_thread` no longer check conf to prune cache, which
has delays and brings a bad experience.
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@xinyiZzz @dataroaring
[opt](memory) Optimize ClearCacheActionimplementation (#38438)
636cf42 
1. `ClearCacheAction` support for specifying cache,
`http://beip:8040/api/clear_cache/all`,
`http://beip:8040/api/clear_cache/DataPageCache`
2. Optimize result returned by `ClearCacheAction`.
3. Optimize Allocator large memory printing stacktrace
4. `cache_prune_stale_thread` no longer check conf to prune cache, which
has delays and brings a bad experience.
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Aug 22, 2024
@xinyiZzz
[opt](memory) Optimize ClearCacheActionimplementation (apache#38438)
497bc98 
1. `ClearCacheAction` support for specifying cache,
`http://beip:8040/api/clear_cache/all`,
`http://beip:8040/api/clear_cache/DataPageCache`
2. Optimize result returned by `ClearCacheAction`.
3. Optimize Allocator large memory printing stacktrace
4. `cache_prune_stale_thread` no longer check conf to prune cache, which
has delays and brings a bad experience.
@xinyiZzz xinyiZzz mentioned this pull request Aug 22, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 22, 2024
@xinyiZzz
[branch-2.1](memory) Optimize ClearCacheActionimplementation (#39796)
1367f74 
pick #38438
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wangbo wangbo wangbo approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@xinyiZzz @doris-robot @yiguolei @wangbo @dataroaring