Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix]Add audit log event queue size limit #37786

Merged
merged 1 commit into from
Jul 16, 2024
Merged

[Fix]Add audit log event queue size limit #37786

merged 1 commit into from
Jul 16, 2024

Conversation

wangbo
Copy link
Contributor

@wangbo wangbo commented Jul 15, 2024

Proposed changes


 num     #instances         #bytes  class name
----------------------------------------------
   1:     375686347    28162104360  [C
   2:      93198001    21621936232  org.apache.doris.plugin.audit.AuditEvent
   3:     375801573    12025650336  java.lang.String
   4:      61002811     2440112440  java.util.LinkedList$Node

We find in some cases excessive audit log may be generatated, so add a queue limit for audit log queue in WorkloadRuntimeStatMgr.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wangbo
Copy link
Contributor Author

wangbo commented Jul 15, 2024

run buildall

@wangbo wangbo force-pushed the 0715_add_limit_for_audit_log branch from b30a8d7 to 8d18194 Compare July 15, 2024 04:06
@wangbo
Copy link
Contributor Author

wangbo commented Jul 15, 2024

run buildall

yiguolei
yiguolei previously approved these changes Jul 15, 2024
View reviewed changes
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 15, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39786 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8d181940e7410099154653992bc9139804b0896d, data reload: false

------ Round 1 ----------------------------------
q1	18216	4427	4382	4382
q2	2016	192	187	187
q3	10452	1242	1089	1089
q4	10201	780	810	780
q5	7577	2668	2680	2668
q6	223	140	136	136
q7	966	596	588	588
q8	9212	2066	2057	2057
q9	8786	6543	6581	6543
q10	8663	3802	3740	3740
q11	438	229	231	229
q12	386	221	223	221
q13	17743	2974	2986	2974
q14	285	238	243	238
q15	519	490	493	490
q16	472	391	381	381
q17	968	635	573	573
q18	8105	7454	7405	7405
q19	8836	1359	1360	1359
q20	712	319	318	318
q21	5086	3146	3998	3146
q22	349	285	282	282
Total cold run time: 120211 ms
Total hot run time: 39786 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4490	4232	4238	4232
q2	375	272	277	272
q3	2989	2771	2791	2771
q4	1866	1629	1627	1627
q5	5330	5369	5299	5299
q6	220	132	132	132
q7	2142	1750	1772	1750
q8	3209	3385	3345	3345
q9	8404	8385	8364	8364
q10	3888	3660	3706	3660
q11	648	486	505	486
q12	771	622	617	617
q13	16376	3000	2986	2986
q14	313	281	278	278
q15	512	488	476	476
q16	480	408	420	408
q17	1768	1508	1469	1469
q18	7728	7748	7402	7402
q19	1727	1424	1535	1424
q20	1998	1776	1773	1773
q21	4811	4736	4734	4734
q22	592	506	488	488
Total cold run time: 70637 ms
Total hot run time: 53993 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171848 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8d181940e7410099154653992bc9139804b0896d, data reload: false

query1	930	380	364	364
query2	6484	1887	1806	1806
query3	6655	213	216	213
query4	23647	17699	17489	17489
query5	4194	502	492	492
query6	289	159	157	157
query7	4593	287	285	285
query8	244	190	194	190
query9	8579	2380	2361	2361
query10	439	271	276	271
query11	12223	10153	10205	10153
query12	129	84	89	84
query13	1668	385	371	371
query14	10388	6838	6518	6518
query15	225	181	168	168
query16	7676	317	320	317
query17	1353	565	549	549
query18	1870	272	269	269
query19	195	150	148	148
query20	87	82	81	81
query21	209	130	123	123
query22	4407	4186	3941	3941
query23	33891	33337	33156	33156
query24	12434	2854	2922	2854
query25	658	360	373	360
query26	1799	150	141	141
query27	3011	268	270	268
query28	7683	1959	1951	1951
query29	1119	647	612	612
query30	286	146	148	146
query31	940	736	782	736
query32	100	55	56	55
query33	770	300	293	293
query34	1005	495	484	484
query35	680	593	568	568
query36	1073	927	949	927
query37	280	77	79	77
query38	2839	2774	2753	2753
query39	853	802	829	802
query40	273	117	118	117
query41	45	44	44	44
query42	118	101	99	99
query43	491	473	466	466
query44	1260	742	730	730
query45	194	162	194	162
query46	1083	712	750	712
query47	1866	1768	1789	1768
query48	374	301	292	292
query49	1199	414	416	414
query50	778	400	387	387
query51	6928	6811	6849	6811
query52	106	99	94	94
query53	359	286	289	286
query54	945	443	448	443
query55	77	73	76	73
query56	290	260	269	260
query57	1176	1041	1025	1025
query58	247	242	269	242
query59	2819	2738	2474	2474
query60	308	313	280	280
query61	98	99	93	93
query62	812	662	653	653
query63	321	295	287	287
query64	10484	2218	1665	1665
query65	3182	3117	3135	3117
query66	1367	339	335	335
query67	15531	15246	14817	14817
query68	5561	539	542	539
query69	616	442	340	340
query70	1189	1180	1190	1180
query71	453	283	282	282
query72	7647	5650	5751	5650
query73	767	330	325	325
query74	6272	5740	5720	5720
query75	3654	2720	2673	2673
query76	3618	917	951	917
query77	656	306	306	306
query78	10686	8939	9705	8939
query79	4727	517	515	515
query80	992	479	483	479
query81	549	226	223	223
query82	677	141	134	134
query83	200	171	169	169
query84	279	91	91	91
query85	1442	396	296	296
query86	468	329	297	297
query87	3265	3129	3221	3129
query88	4715	2449	2460	2449
query89	499	385	393	385
query90	1887	194	201	194
query91	128	100	98	98
query92	66	49	51	49
query93	3505	493	496	493
query94	1164	213	215	213
query95	398	315	323	315
query96	610	274	274	274
query97	3214	3083	3045	3045
query98	220	199	196	196
query99	1696	1284	1254	1254
Total cold run time: 290456 ms
Total hot run time: 171848 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8d181940e7410099154653992bc9139804b0896d, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.69	0.07	0.07
query5	0.49	0.49	0.48
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.54	0.49	0.50
query10	0.54	0.53	0.55
query11	0.16	0.12	0.12
query12	0.15	0.12	0.13
query13	0.59	0.59	0.58
query14	0.77	0.79	0.77
query15	0.86	0.81	0.81
query16	0.36	0.36	0.36
query17	1.01	0.98	1.01
query18	0.23	0.22	0.22
query19	1.88	1.77	1.81
query20	0.01	0.01	0.01
query21	15.42	0.74	0.65
query22	3.90	7.55	1.77
query23	18.22	1.38	1.25
query24	2.09	0.23	0.21
query25	0.15	0.10	0.08
query26	0.30	0.20	0.21
query27	0.46	0.23	0.22
query28	13.39	1.01	0.99
query29	12.68	3.30	3.27
query30	0.26	0.06	0.05
query31	2.87	0.39	0.39
query32	3.27	0.49	0.47
query33	2.89	2.95	2.87
query34	17.15	4.37	4.33
query35	4.40	4.35	4.47
query36	0.65	0.45	0.46
query37	0.19	0.16	0.16
query38	0.16	0.15	0.16
query39	0.04	0.04	0.03
query40	0.15	0.13	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 109.66 s
Total hot run time: 30.37 s

@wangbo
Copy link
Contributor Author

wangbo commented Jul 15, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 15, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 39977 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c08529dcc5145d19fffcd0127fd136662039184c, data reload: false

------ Round 1 ----------------------------------
q1	17606	4407	4366	4366
q2	2023	193	190	190
q3	10458	1183	1139	1139
q4	10196	737	835	737
q5	7571	2702	2681	2681
q6	229	138	140	138
q7	943	589	593	589
q8	9230	2088	2063	2063
q9	8644	6536	6522	6522
q10	8661	3726	3785	3726
q11	466	235	243	235
q12	394	225	228	225
q13	18698	2996	3012	2996
q14	282	238	231	231
q15	527	492	484	484
q16	503	386	379	379
q17	955	655	702	655
q18	8068	7602	7352	7352
q19	7204	1416	1447	1416
q20	698	317	331	317
q21	4914	3302	3249	3249
q22	353	287	287	287
Total cold run time: 118623 ms
Total hot run time: 39977 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4463	4281	4299	4281
q2	378	271	269	269
q3	3042	2972	2955	2955
q4	1999	1755	1774	1755
q5	5588	5534	5460	5460
q6	233	134	141	134
q7	2291	1875	1888	1875
q8	3292	3506	3372	3372
q9	8765	8798	8817	8798
q10	4190	3778	3915	3778
q11	628	501	515	501
q12	818	643	655	643
q13	16016	3158	3227	3158
q14	327	279	311	279
q15	547	491	490	490
q16	505	442	445	442
q17	1807	1548	1511	1511
q18	8221	7918	7821	7821
q19	1784	1521	1637	1521
q20	2139	1883	1873	1873
q21	8037	4753	4749	4749
q22	599	515	530	515
Total cold run time: 75669 ms
Total hot run time: 56180 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173995 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c08529dcc5145d19fffcd0127fd136662039184c, data reload: false

query1	913	376	362	362
query2	6442	1891	1824	1824
query3	6637	206	214	206
query4	25576	17517	17607	17517
query5	3779	478	467	467
query6	259	190	172	172
query7	4568	282	288	282
query8	248	189	202	189
query9	8440	2371	2357	2357
query10	430	275	269	269
query11	11809	10139	10115	10115
query12	124	86	82	82
query13	1662	378	368	368
query14	10098	7819	7940	7819
query15	235	177	169	169
query16	7349	328	318	318
query17	1792	560	553	553
query18	1838	287	281	281
query19	201	154	156	154
query20	92	88	86	86
query21	206	140	129	129
query22	4435	4181	4053	4053
query23	33966	33665	33686	33665
query24	10897	2984	2880	2880
query25	669	415	408	408
query26	1010	158	154	154
query27	2466	275	281	275
query28	6923	2064	2043	2043
query29	917	662	643	643
query30	255	156	158	156
query31	960	832	782	782
query32	102	54	59	54
query33	780	324	308	308
query34	973	488	509	488
query35	700	589	670	589
query36	1142	964	988	964
query37	167	87	84	84
query38	2969	2920	2908	2908
query39	884	863	867	863
query40	212	125	119	119
query41	49	48	45	45
query42	121	101	99	99
query43	493	468	456	456
query44	1222	732	732	732
query45	192	159	162	159
query46	1096	722	712	712
query47	1883	1770	1751	1751
query48	370	294	295	294
query49	845	405	406	405
query50	788	388	390	388
query51	6913	6841	6735	6735
query52	104	95	92	92
query53	358	288	297	288
query54	905	461	440	440
query55	73	72	72	72
query56	281	259	264	259
query57	1090	1050	1084	1050
query58	241	256	249	249
query59	2769	2637	2666	2637
query60	294	275	269	269
query61	98	91	95	91
query62	781	641	633	633
query63	324	288	284	284
query64	9681	2207	1666	1666
query65	3173	3192	3109	3109
query66	767	319	328	319
query67	15587	14874	14935	14874
query68	6233	526	530	526
query69	762	413	374	374
query70	1250	1166	1166	1166
query71	509	281	273	273
query72	8359	5453	5814	5453
query73	799	325	321	321
query74	6068	5639	5683	5639
query75	4803	2646	2665	2646
query76	4453	977	878	878
query77	745	298	294	294
query78	9621	8982	8924	8924
query79	7860	521	515	515
query80	1062	478	482	478
query81	580	222	220	220
query82	745	137	138	137
query83	265	219	166	166
query84	274	86	94	86
query85	1385	312	305	305
query86	419	317	325	317
query87	3371	3132	3146	3132
query88	4970	2461	2475	2461
query89	497	383	384	383
query90	2043	202	194	194
query91	128	100	102	100
query92	66	50	51	50
query93	6070	507	499	499
query94	1317	212	210	210
query95	444	323	315	315
query96	600	282	272	272
query97	3205	2988	3008	2988
query98	221	205	192	192
query99	1515	1268	1218	1218
Total cold run time: 294262 ms
Total hot run time: 173995 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.29 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c08529dcc5145d19fffcd0127fd136662039184c, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.03
query3	0.22	0.05	0.05
query4	1.68	0.06	0.08
query5	0.50	0.48	0.49
query6	1.14	0.71	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.50	0.49
query10	0.53	0.55	0.55
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.76	0.78	0.77
query15	0.85	0.82	0.82
query16	0.36	0.38	0.37
query17	1.03	1.03	1.03
query18	0.23	0.22	0.22
query19	1.91	1.79	1.65
query20	0.01	0.01	0.00
query21	15.39	0.74	0.65
query22	4.73	6.62	1.69
query23	18.29	1.43	1.21
query24	2.13	0.23	0.22
query25	0.15	0.09	0.08
query26	0.29	0.20	0.21
query27	0.46	0.23	0.22
query28	13.27	1.01	1.00
query29	12.64	3.34	3.30
query30	0.25	0.05	0.05
query31	2.87	0.40	0.40
query32	3.26	0.48	0.47
query33	2.87	2.85	2.92
query34	16.96	4.43	4.38
query35	4.38	4.40	4.45
query36	0.66	0.46	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.10	0.05	0.06
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.2 s
Total hot run time: 30.29 s

@wangbo
Copy link
Contributor Author

wangbo commented Jul 15, 2024

run buildall

1 similar comment
@wangbo
Copy link
Contributor Author

wangbo commented Jul 15, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39587 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f8ffed319fe54f3a3bf047bea8be8ff16e805c14, data reload: false

------ Round 1 ----------------------------------
q1	17627	4528	4277	4277
q2	2022	188	186	186
q3	10451	1266	1009	1009
q4	10194	802	757	757
q5	7594	2689	2825	2689
q6	224	142	143	142
q7	986	607	616	607
q8	9570	2085	2076	2076
q9	9221	6530	6523	6523
q10	8818	3800	3789	3789
q11	466	238	234	234
q12	405	231	225	225
q13	17764	2969	2973	2969
q14	277	241	237	237
q15	517	483	497	483
q16	495	389	375	375
q17	948	620	633	620
q18	8195	7493	7336	7336
q19	8135	1339	1305	1305
q20	656	326	331	326
q21	4865	3144	3204	3144
q22	354	278	281	278
Total cold run time: 119784 ms
Total hot run time: 39587 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4448	4258	4199	4199
q2	383	267	268	267
q3	3008	2719	2707	2707
q4	1892	1651	1584	1584
q5	5283	5303	5296	5296
q6	220	134	135	134
q7	2099	1736	1684	1684
q8	3192	3361	3297	3297
q9	8352	8356	8373	8356
q10	3913	3694	3653	3653
q11	568	481	490	481
q12	771	592	623	592
q13	17544	2958	2992	2958
q14	310	273	277	273
q15	521	476	475	475
q16	470	418	426	418
q17	1769	1486	1459	1459
q18	7655	7500	7330	7330
q19	1646	1556	1561	1556
q20	1992	1794	1789	1789
q21	4738	4617	4719	4617
q22	547	493	475	475
Total cold run time: 71321 ms
Total hot run time: 53600 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172751 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f8ffed319fe54f3a3bf047bea8be8ff16e805c14, data reload: false

query1	921	370	364	364
query2	6455	1942	1888	1888
query3	6671	206	217	206
query4	28421	17680	17354	17354
query5	4294	488	477	477
query6	277	164	168	164
query7	4584	281	278	278
query8	239	203	200	200
query9	8414	2343	2334	2334
query10	459	286	279	279
query11	12103	10048	9830	9830
query12	134	85	84	84
query13	1646	362	371	362
query14	10097	7750	7621	7621
query15	224	165	174	165
query16	7797	324	325	324
query17	1787	578	547	547
query18	1989	285	282	282
query19	191	160	158	158
query20	95	84	83	83
query21	205	127	124	124
query22	4382	4172	3991	3991
query23	33830	33326	33008	33008
query24	12011	2888	2820	2820
query25	661	388	394	388
query26	1775	151	149	149
query27	2984	267	268	267
query28	7616	1952	1943	1943
query29	1079	632	623	623
query30	290	152	149	149
query31	972	751	780	751
query32	92	55	58	55
query33	793	307	313	307
query34	954	495	501	495
query35	695	570	582	570
query36	1121	940	921	921
query37	282	80	85	80
query38	2875	2768	2768	2768
query39	872	824	845	824
query40	286	126	122	122
query41	50	46	48	46
query42	120	98	103	98
query43	521	498	482	482
query44	1248	732	731	731
query45	195	164	160	160
query46	1083	733	729	729
query47	1849	1807	1797	1797
query48	373	303	292	292
query49	1211	420	424	420
query50	765	399	401	399
query51	6891	6826	6699	6699
query52	105	97	94	94
query53	359	286	284	284
query54	940	454	457	454
query55	78	75	77	75
query56	318	293	301	293
query57	1147	1100	1071	1071
query58	277	349	245	245
query59	2836	2762	2582	2582
query60	301	277	291	277
query61	99	96	107	96
query62	857	657	663	657
query63	323	297	292	292
query64	10477	2232	1672	1672
query65	3194	3101	3121	3101
query66	1406	344	349	344
query67	15568	14982	14919	14919
query68	9016	550	556	550
query69	730	446	375	375
query70	1435	1136	1145	1136
query71	535	274	278	274
query72	9001	5877	5757	5757
query73	2252	326	324	324
query74	6193	5655	5630	5630
query75	4631	2726	2718	2718
query76	5374	899	920	899
query77	725	317	309	309
query78	10605	8905	9051	8905
query79	7814	518	522	518
query80	1453	488	478	478
query81	574	216	224	216
query82	287	127	144	127
query83	239	167	164	164
query84	279	88	91	88
query85	971	334	301	301
query86	350	306	335	306
query87	3300	3131	3113	3113
query88	4877	2441	2460	2441
query89	522	377	375	375
query90	2027	197	196	196
query91	130	101	100	100
query92	64	49	49	49
query93	5664	507	489	489
query94	1193	219	214	214
query95	400	317	314	314
query96	613	273	274	273
query97	3189	3046	3012	3012
query98	218	201	198	198
query99	1570	1238	1266	1238
Total cold run time: 309395 ms
Total hot run time: 172751 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.5 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f8ffed319fe54f3a3bf047bea8be8ff16e805c14, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.03	0.03
query3	0.23	0.04	0.04
query4	1.68	0.06	0.07
query5	0.48	0.50	0.47
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.48
query10	0.54	0.54	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.13
query13	0.59	0.58	0.59
query14	0.76	0.77	0.78
query15	0.86	0.82	0.82
query16	0.37	0.36	0.34
query17	1.01	0.98	1.04
query18	0.22	0.21	0.20
query19	1.80	1.66	1.66
query20	0.01	0.00	0.00
query21	15.40	0.73	0.64
query22	4.60	7.30	1.94
query23	18.30	1.45	1.31
query24	2.09	0.22	0.22
query25	0.15	0.09	0.09
query26	0.30	0.21	0.20
query27	0.45	0.23	0.23
query28	13.30	1.03	1.00
query29	12.58	3.24	3.26
query30	0.25	0.06	0.07
query31	2.85	0.38	0.38
query32	3.29	0.47	0.47
query33	2.87	2.85	2.95
query34	16.99	4.39	4.41
query35	4.47	4.41	4.43
query36	0.65	0.47	0.47
query37	0.19	0.16	0.16
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.13	0.12
query41	0.09	0.05	0.04
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 110 s
Total hot run time: 30.5 s

yiguolei
yiguolei previously approved these changes Jul 15, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 15, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@yiguolei yiguolei added the usercase Important user case type label label Jul 16, 2024
@wangbo wangbo force-pushed the 0715_add_limit_for_audit_log branch from f8ffed3 to c6616ff Compare July 16, 2024 03:46
@wangbo
Copy link
Contributor Author

wangbo commented Jul 16, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 16, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 39942 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c6616ff11d4fd7df75b1a6b176e2802b942dd49e, data reload: false

------ Round 1 ----------------------------------
q1	17865	4481	4358	4358
q2	2461	197	187	187
q3	11181	1183	1112	1112
q4	10784	844	882	844
q5	7547	2722	2696	2696
q6	227	140	138	138
q7	972	618	606	606
q8	9410	2047	2056	2047
q9	8604	6511	6564	6511
q10	8668	3728	3755	3728
q11	453	230	234	230
q12	390	219	218	218
q13	18914	2983	2987	2983
q14	278	241	244	241
q15	524	476	462	462
q16	468	380	383	380
q17	967	659	675	659
q18	8137	7448	7336	7336
q19	5904	1441	1385	1385
q20	662	326	324	324
q21	4848	3287	3213	3213
q22	358	284	290	284
Total cold run time: 119622 ms
Total hot run time: 39942 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4373	4259	4244	4244
q2	357	269	263	263
q3	2975	2732	2734	2732
q4	1882	1610	1560	1560
q5	5283	5319	5302	5302
q6	225	128	131	128
q7	2122	1756	1667	1667
q8	3205	3314	3325	3314
q9	8407	8412	8399	8399
q10	3890	3753	3732	3732
q11	571	480	496	480
q12	766	600	595	595
q13	16759	2972	2981	2972
q14	289	263	258	258
q15	532	492	483	483
q16	465	421	442	421
q17	1794	1512	1465	1465
q18	7544	7473	7511	7473
q19	1657	1667	1648	1648
q20	2031	1807	1791	1791
q21	4914	4627	4769	4627
q22	604	474	477	474
Total cold run time: 70645 ms
Total hot run time: 54028 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172948 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c6616ff11d4fd7df75b1a6b176e2802b942dd49e, data reload: false

query1	909	377	364	364
query2	6464	1852	1810	1810
query3	6651	207	224	207
query4	28427	17566	18016	17566
query5	4184	475	483	475
query6	300	183	163	163
query7	4590	304	283	283
query8	249	208	203	203
query9	8471	2427	2401	2401
query10	464	282	271	271
query11	11691	10159	10163	10159
query12	132	86	86	86
query13	1669	366	362	362
query14	10113	7649	7749	7649
query15	215	170	167	167
query16	7613	307	300	300
query17	1841	539	520	520
query18	1812	285	268	268
query19	189	146	151	146
query20	89	81	81	81
query21	209	129	124	124
query22	4375	4037	4060	4037
query23	33619	32945	33180	32945
query24	11069	2844	2824	2824
query25	627	369	391	369
query26	1563	147	147	147
query27	2898	273	277	273
query28	7308	2034	2029	2029
query29	982	629	606	606
query30	284	151	145	145
query31	936	729	757	729
query32	97	53	54	53
query33	764	289	287	287
query34	958	476	485	476
query35	689	563	578	563
query36	1099	930	940	930
query37	153	80	80	80
query38	2859	2781	2747	2747
query39	878	819	822	819
query40	276	121	120	120
query41	48	47	44	44
query42	120	96	136	96
query43	490	464	474	464
query44	1162	741	721	721
query45	188	162	164	162
query46	1080	744	713	713
query47	1850	1768	1762	1762
query48	363	298	295	295
query49	1171	413	408	408
query50	782	400	399	399
query51	6911	6865	6837	6837
query52	103	96	92	92
query53	360	300	301	300
query54	914	445	453	445
query55	74	76	75	75
query56	290	259	263	259
query57	1167	1055	1047	1047
query58	242	247	261	247
query59	2938	2557	2680	2557
query60	293	271	325	271
query61	96	95	94	94
query62	816	634	643	634
query63	327	292	287	287
query64	10456	2221	1625	1625
query65	3146	3097	3088	3088
query66	1366	333	336	333
query67	15301	15079	15125	15079
query68	4606	563	591	563
query69	463	340	317	317
query70	1195	1096	1149	1096
query71	405	277	278	277
query72	6969	5605	5773	5605
query73	750	336	326	326
query74	6075	5618	5683	5618
query75	3394	2699	2708	2699
query76	2717	931	919	919
query77	459	324	318	318
query78	9441	8949	8939	8939
query79	2126	522	509	509
query80	1418	485	471	471
query81	563	220	270	220
query82	739	131	130	130
query83	231	163	164	163
query84	244	84	91	84
query85	2046	306	307	306
query86	492	307	317	307
query87	3240	3094	3125	3094
query88	4280	2404	2358	2358
query89	488	397	384	384
query90	1854	196	196	196
query91	129	100	101	100
query92	62	52	54	52
query93	2438	515	515	515
query94	1283	219	214	214
query95	402	306	313	306
query96	600	270	268	268
query97	3228	3018	3019	3018
query98	224	249	194	194
query99	1564	1264	1305	1264
Total cold run time: 283895 ms
Total hot run time: 172948 ms

@wangbo wangbo force-pushed the 0715_add_limit_for_audit_log branch from c6616ff to dabb4ac Compare July 16, 2024 04:21
@wangbo
Copy link
Contributor Author

wangbo commented Jul 16, 2024

run buildall

yiguolei
yiguolei approved these changes Jul 16, 2024
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 16, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39975 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dabb4acc987a4fe56bd3dc9be4026aec9f44f8fd, data reload: false

------ Round 1 ----------------------------------
q1	17643	4622	4296	4296
q2	2014	190	186	186
q3	10544	1327	1156	1156
q4	10252	757	859	757
q5	7557	2722	2652	2652
q6	226	140	140	140
q7	1034	625	593	593
q8	9217	2097	2118	2097
q9	8727	6592	6571	6571
q10	8647	3765	3775	3765
q11	458	236	232	232
q12	393	229	231	229
q13	17758	2965	2946	2946
q14	276	249	241	241
q15	525	474	486	474
q16	496	395	380	380
q17	977	727	687	687
q18	8108	7626	7428	7428
q19	8002	1475	1343	1343
q20	708	330	313	313
q21	5042	3208	3208	3208
q22	352	292	281	281
Total cold run time: 118956 ms
Total hot run time: 39975 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4403	4311	4244	4244
q2	388	264	280	264
q3	3008	2789	2728	2728
q4	1819	1611	1616	1611
q5	5272	5337	5293	5293
q6	222	135	135	135
q7	2143	1697	1764	1697
q8	3227	3361	3315	3315
q9	8437	8407	8379	8379
q10	3869	3709	3679	3679
q11	587	485	498	485
q12	777	604	607	604
q13	16690	2977	2994	2977
q14	294	279	275	275
q15	519	480	477	477
q16	478	416	417	416
q17	1813	1494	1472	1472
q18	7707	7572	7436	7436
q19	1688	1643	1566	1566
q20	1988	1794	1787	1787
q21	9323	4683	4698	4683
q22	574	507	503	503
Total cold run time: 75226 ms
Total hot run time: 54026 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172785 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dabb4acc987a4fe56bd3dc9be4026aec9f44f8fd, data reload: false

query1	938	373	366	366
query2	6467	1926	1822	1822
query3	6654	205	217	205
query4	28448	17362	17196	17196
query5	4215	504	510	504
query6	280	180	180	180
query7	4603	284	288	284
query8	242	200	193	193
query9	8405	2405	2401	2401
query10	453	276	274	274
query11	10761	9967	10202	9967
query12	133	87	81	81
query13	1638	371	360	360
query14	10073	6964	7716	6964
query15	230	166	168	166
query16	7579	329	320	320
query17	1343	539	562	539
query18	1923	282	278	278
query19	204	157	157	157
query20	90	77	84	77
query21	206	130	127	127
query22	4306	3999	3936	3936
query23	33590	33163	33496	33163
query24	12374	2864	2965	2864
query25	682	396	400	396
query26	1784	152	148	148
query27	3006	269	277	269
query28	7499	2040	2033	2033
query29	1126	659	644	644
query30	289	154	152	152
query31	1017	741	761	741
query32	103	56	58	56
query33	790	324	313	313
query34	950	499	506	499
query35	676	595	584	584
query36	1088	959	944	944
query37	277	91	81	81
query38	2837	2728	2733	2728
query39	880	828	809	809
query40	283	124	126	124
query41	49	49	46	46
query42	122	102	103	102
query43	498	494	457	457
query44	1299	730	731	730
query45	198	162	162	162
query46	1195	740	738	738
query47	1831	1776	1792	1776
query48	372	310	299	299
query49	1183	427	427	427
query50	799	412	402	402
query51	6931	6791	6754	6754
query52	107	94	96	94
query53	366	289	297	289
query54	913	459	457	457
query55	77	78	76	76
query56	321	284	298	284
query57	1154	1067	1055	1055
query58	277	261	271	261
query59	2824	2708	2755	2708
query60	319	303	315	303
query61	119	118	124	118
query62	809	664	663	663
query63	332	299	295	295
query64	10650	2313	1760	1760
query65	3193	3163	3107	3107
query66	1243	352	343	343
query67	15690	14978	15005	14978
query68	6327	552	547	547
query69	697	491	360	360
query70	1197	1167	1154	1154
query71	504	416	280	280
query72	9059	5753	5657	5657
query73	795	326	329	326
query74	6030	5669	5625	5625
query75	4725	2645	2677	2645
query76	4676	967	942	942
query77	765	307	317	307
query78	9821	9024	9673	9024
query79	9696	536	544	536
query80	1123	476	485	476
query81	584	220	224	220
query82	696	145	133	133
query83	335	170	172	170
query84	269	85	88	85
query85	1518	352	299	299
query86	433	312	318	312
query87	3293	3100	3185	3100
query88	5285	2385	2395	2385
query89	513	388	390	388
query90	2112	203	201	201
query91	134	101	100	100
query92	62	51	50	50
query93	6763	522	514	514
query94	1438	223	215	215
query95	399	321	318	318
query96	617	274	271	271
query97	3127	3087	3064	3064
query98	229	202	196	196
query99	1567	1287	1256	1256
Total cold run time: 306102 ms
Total hot run time: 172785 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dabb4acc987a4fe56bd3dc9be4026aec9f44f8fd, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.09	0.09
query5	0.50	0.49	0.48
query6	1.17	0.72	0.73
query7	0.02	0.02	0.02
query8	0.06	0.05	0.04
query9	0.55	0.49	0.49
query10	0.55	0.56	0.54
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.59	0.60	0.59
query14	0.75	0.79	0.78
query15	0.86	0.82	0.81
query16	0.36	0.35	0.36
query17	0.96	1.01	1.01
query18	0.23	0.22	0.22
query19	1.89	1.65	1.66
query20	0.01	0.01	0.01
query21	15.41	0.74	0.65
query22	4.37	7.80	1.82
query23	18.25	1.34	1.29
query24	2.11	0.23	0.22
query25	0.16	0.09	0.08
query26	0.30	0.21	0.21
query27	0.45	0.23	0.22
query28	13.27	1.01	1.00
query29	12.58	3.32	3.28
query30	0.26	0.07	0.05
query31	2.88	0.39	0.39
query32	3.26	0.48	0.47
query33	2.89	2.87	2.87
query34	17.02	4.34	4.34
query35	4.39	4.40	4.45
query36	0.64	0.47	0.47
query37	0.18	0.15	0.16
query38	0.16	0.14	0.15
query39	0.05	0.04	0.03
query40	0.15	0.12	0.12
query41	0.08	0.04	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.76 s
Total hot run time: 30.48 s

xinyiZzz
xinyiZzz approved these changes Jul 16, 2024
View reviewed changes
Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wangbo wangbo merged commit 9032179 into apache:master Jul 16, 2024
26 of 29 checks passed
@wangbo wangbo mentioned this pull request Jul 16, 2024
Merged
yiguolei pushed a commit that referenced this pull request Jul 16, 2024
@wangbo
[pick]Add audit log event queue size limit (#37914)
81a7542 
## Proposed changes
pick #37786
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@wangbo @seawinde
[Fix]Add audit log event queue size limit (apache#37786)
938a3bc 
## Proposed changes
```

 num     #instances         #bytes  class name
----------------------------------------------
   1:     375686347    28162104360  [C
   2:      93198001    21621936232  org.apache.doris.plugin.audit.AuditEvent
   3:     375801573    12025650336  java.lang.String
   4:      6100281     2440112440  java.util.LinkedList$Node
```
We find in some cases excessive audit log may be generatated, so add a
queue limit for audit log queue in WorkloadRuntimeStatMgr.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@wangbo @dataroaring
[Fix]Add audit log event queue size limit (#37786)
cc29fd1 
## Proposed changes
```

 num     #instances         #bytes  class name
----------------------------------------------
   1:     375686347    28162104360  [C
   2:      93198001    21621936232  org.apache.doris.plugin.audit.AuditEvent
   3:     375801573    12025650336  java.lang.String
   4:      6100281     2440112440  java.util.LinkedList$Node
```
We find in some cases excessive audit log may be generatated, so add a
queue limit for audit log queue in WorkloadRuntimeStatMgr.
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
Open
1 task
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xinyiZzz xinyiZzz xinyiZzz approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@wangbo @doris-robot @yiguolei @xinyiZzz @dataroaring