Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(feature)(LOG) Support to change be vlog level dynamically using http #28785

Merged
merged 5 commits into from
Jan 6, 2024

Conversation

ByteYue
Copy link
Contributor

@ByteYue ByteYue commented Dec 21, 2023

Proposed changes

Issue Number: close #xxx
Now we can adjust be's vlog level dynamically using http action. According to 649 and 650. we just need to use SetVLOGLevel to update the log level of the corresponding module.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

// specific language governing permissions and limitations
// under the License.

#include <glog/vlog_is_on.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'glog/vlog_is_on.h' file not found [clang-diagnostic-error]

#include <glog/vlog_is_on.h>
         ^

dataroaring
dataroaring previously approved these changes Dec 21, 2023
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Dec 21, 2023
Copy link
Contributor

PR approved by anyone and no changes requested.

@ByteYue
Copy link
Contributor Author

ByteYue commented Dec 22, 2023

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.48% (8539/23408)
Line Coverage: 28.59% (69418/242823)
Region Coverage: 27.61% (35920/130110)
Branch Coverage: 24.34% (18355/75398)
Coverage Report: http://coverage.selectdb-in.cc/coverage/60ac72931d4840259b7594cba294a8e0c6b579eb_60ac72931d4840259b7594cba294a8e0c6b579eb/report/index.html

@ByteYue
Copy link
Contributor Author

ByteYue commented Dec 25, 2023

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 25, 2023
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.58% (8556/23392)
Line Coverage: 28.65% (69559/242806)
Region Coverage: 27.66% (35980/130093)
Branch Coverage: 24.38% (18390/75416)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4e6cedcaf448e2807d377dabc62e66ff3abe2916_4e6cedcaf448e2807d377dabc62e66ff3abe2916/report/index.html

@ByteYue
Copy link
Contributor Author

ByteYue commented Dec 25, 2023

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit fa1b471c3ef508cc9bda6d5d342753cffca7e8e2, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4701	4443	4451	4443
q2	394	166	156	156
q3	1465	1256	1298	1256
q4	1122	928	925	925
q5	3173	3192	3161	3161
q6	250	131	133	131
q7	1010	498	505	498
q8	2172	2220	2211	2211
q9	6682	6662	6653	6653
q10	3233	3271	3283	3271
q11	309	189	193	189
q12	358	207	212	207
q13	4605	3753	3807	3753
q14	244	212	218	212
q15	575	523	527	523
q16	444	383	384	383
q17	1008	652	593	593
q18	7045	6860	6789	6789
q19	1529	1437	1413	1413
q20	510	286	332	286
q21	3084	2695	2628	2628
q22	343	282	282	282
Total cold run time: 44256 ms
Total hot run time: 39963 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4346	4337	4335	4335
q2	269	166	173	166
q3	3499	3498	3492	3492
q4	2405	2369	2387	2369
q5	5694	5716	5722	5716
q6	244	121	122	121
q7	2354	1846	1848	1846
q8	3528	3542	3523	3523
q9	8987	9065	8959	8959
q10	3912	3982	4013	3982
q11	476	362	363	362
q12	767	591	594	591
q13	4261	3607	3551	3551
q14	291	259	258	258
q15	578	518	527	518
q16	488	435	474	435
q17	1870	1859	1848	1848
q18	8512	8112	8107	8107
q19	1729	1739	1740	1739
q20	2243	1949	1931	1931
q21	6499	6147	6143	6143
q22	498	418	426	418
Total cold run time: 63450 ms
Total hot run time: 60410 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.58% (8556/23392)
Line Coverage: 28.65% (69563/242806)
Region Coverage: 27.65% (35977/130093)
Branch Coverage: 24.38% (18388/75416)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fa1b471c3ef508cc9bda6d5d342753cffca7e8e2_fa1b471c3ef508cc9bda6d5d342753cffca7e8e2/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.35 seconds
stream load tsv: 581 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 37.8 seconds inserted 10000000 Rows, about 264K ops/s
storage size: 17183919436 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit fa1b471c3ef508cc9bda6d5d342753cffca7e8e2, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4706	4398	4403	4398
q2	364	152	159	152
q3	1470	1284	1238	1238
q4	1106	916	912	912
q5	3134	3185	3172	3172
q6	249	132	131	131
q7	986	498	481	481
q8	2185	2228	2310	2228
q9	6693	6687	6645	6645
q10	3210	3298	3293	3293
q11	306	190	186	186
q12	355	216	215	215
q13	4530	3802	3790	3790
q14	244	212	215	212
q15	577	523	536	523
q16	436	390	384	384
q17	1015	637	560	560
q18	7039	6797	6724	6724
q19	1517	1399	1423	1399
q20	500	318	329	318
q21	3065	2647	2686	2647
q22	352	289	279	279
Total cold run time: 44039 ms
Total hot run time: 39887 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4377	4322	4367	4322
q2	269	171	170	170
q3	3506	3496	3501	3496
q4	2409	2383	2388	2383
q5	5733	5744	5710	5710
q6	242	124	124	124
q7	2369	1865	1874	1865
q8	3518	3538	3520	3520
q9	9033	9047	8951	8951
q10	3947	4020	4008	4008
q11	486	365	366	365
q12	766	592	614	592
q13	4287	3577	3564	3564
q14	282	261	253	253
q15	575	529	523	523
q16	506	441	492	441
q17	1891	1819	1838	1819
q18	8615	8165	8094	8094
q19	1746	1767	1732	1732
q20	2256	1937	1936	1936
q21	6519	6182	6145	6145
q22	500	429	423	423
Total cold run time: 63832 ms
Total hot run time: 60436 ms

@ByteYue ByteYue force-pushed the add_http_to_control_be_log_level branch from fa1b471 to edd58b1 Compare December 27, 2023 05:36
@ByteYue
Copy link
Contributor Author

ByteYue commented Dec 27, 2023

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.61% (8587/23453)
Line Coverage: 28.68% (69798/243386)
Region Coverage: 27.64% (36064/130464)
Branch Coverage: 24.37% (18425/75620)
Coverage Report: http://coverage.selectdb-in.cc/coverage/edd58b17b64950c9d589fbc9ff862dae7d360e43_edd58b17b64950c9d589fbc9ff862dae7d360e43/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit edd58b17b64950c9d589fbc9ff862dae7d360e43, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4744	4471	4472	4471
q2	364	141	159	141
q3	1470	1287	1213	1213
q4	1158	965	899	899
q5	3154	3152	3175	3152
q6	259	137	138	137
q7	1048	500	475	475
q8	2209	2276	2253	2253
q9	6704	6700	6663	6663
q10	3223	3295	3298	3295
q11	327	203	210	203
q12	353	202	206	202
q13	4521	3778	3772	3772
q14	246	216	213	213
q15	586	526	527	526
q16	443	388	382	382
q17	1048	706	492	492
q18	7116	6859	6869	6859
q19	1636	1646	1627	1627
q20	565	323	299	299
q21	3280	2796	2749	2749
q22	362	303	309	303
Total cold run time: 44816 ms
Total hot run time: 40326 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4393	4401	4370	4370
q2	269	166	169	166
q3	3522	3512	3500	3500
q4	2439	2419	2412	2412
q5	5743	5735	5737	5735
q6	251	127	127	127
q7	2370	1811	1868	1811
q8	3606	3600	3609	3600
q9	9023	8987	9012	8987
q10	3895	3993	3984	3984
q11	489	400	386	386
q12	765	589	605	589
q13	4295	3552	3554	3552
q14	289	250	268	250
q15	566	524	521	521
q16	505	449	454	449
q17	1974	1947	1941	1941
q18	8607	8294	8084	8084
q19	1850	1835	1839	1835
q20	2282	1949	1937	1937
q21	6626	6278	6275	6275
q22	548	455	449	449
Total cold run time: 64307 ms
Total hot run time: 60960 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.68 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17184064774 Bytes

@ByteYue
Copy link
Contributor Author

ByteYue commented Dec 28, 2023

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.60% (8589/23467)
Line Coverage: 28.67% (69831/243579)
Region Coverage: 27.66% (36122/130581)
Branch Coverage: 24.39% (18466/75722)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2a508884f37452048d8b7dda580f08c9d1b85df3_2a508884f37452048d8b7dda580f08c9d1b85df3/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 2a508884f37452048d8b7dda580f08c9d1b85df3, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5045	4642	4668	4642
q2	365	160	139	139
q3	1477	1286	1198	1198
q4	1119	893	871	871
q5	3200	3165	3180	3165
q6	249	127	127	127
q7	1049	492	495	492
q8	2272	2263	2247	2247
q9	6773	6689	6667	6667
q10	3202	3285	3278	3278
q11	337	214	215	214
q12	346	204	211	204
q13	4168	3449	3427	3427
q14	243	218	217	217
q15	583	521	522	521
q16	437	402	386	386
q17	1050	780	580	580
q18	7059	6810	6901	6810
q19	1616	1634	1654	1634
q20	565	294	312	294
q21	3154	2703	2722	2703
q22	368	297	315	297
Total cold run time: 44677 ms
Total hot run time: 40113 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4584	4586	4563	4563
q2	275	170	164	164
q3	3373	3366	3356	3356
q4	2224	2197	2198	2197
q5	5702	5702	5712	5702
q6	244	118	118	118
q7	2379	1899	1847	1847
q8	3598	3608	3616	3608
q9	9034	8952	8910	8910
q10	3791	3892	3908	3892
q11	482	373	374	373
q12	768	590	620	590
q13	3907	3210	3153	3153
q14	286	246	259	246
q15	575	521	519	519
q16	500	446	454	446
q17	1972	1976	1951	1951
q18	8686	8284	8303	8284
q19	1916	1753	1763	1753
q20	2257	1935	1915	1915
q21	6149	5743	5753	5743
q22	557	482	446	446
Total cold run time: 63259 ms
Total hot run time: 59776 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.1 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17183979427 Bytes

@ByteYue ByteYue force-pushed the add_http_to_control_be_log_level branch from 2a50888 to c10c1d1 Compare January 2, 2024 05:52
@ByteYue
Copy link
Contributor Author

ByteYue commented Jan 2, 2024

run buildall

Copy link
Contributor

github-actions bot commented Jan 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit c10c1d1da590c6ddbe25719e54e8ac4368fec265, data reload: false

------ Round 1 ----------------------------------
q1	17741	5064	5074	5064
q2	2011	157	148	148
q3	10534	1110	1106	1106
q4	10169	811	848	811
q5	7813	2975	2972	2972
q6	213	143	138	138
q7	918	561	507	507
q8	9296	2014	2018	2014
q9	6816	6459	6412	6412
q10	8243	3048	3064	3048
q11	450	217	222	217
q12	394	233	229	229
q13	17986	3636	3635	3635
q14	239	215	219	215
q15	582	552	543	543
q16	458	389	387	387
q17	966	451	502	451
q18	7185	6658	6671	6658
q19	1573	1330	1305	1305
q20	697	333	343	333
q21	2816	2347	2439	2347
q22	391	338	337	337
Total cold run time: 107491 ms
Total hot run time: 38877 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5129	5085	5135	5085
q2	339	252	254	252
q3	3296	3303	3260	3260
q4	2102	2005	2005	2005
q5	5813	5803	5779	5779
q6	218	128	127	127
q7	2317	1867	1877	1867
q8	3349	3454	3473	3454
q9	8825	8827	8736	8736
q10	3809	3842	3828	3828
q11	588	491	524	491
q12	800	652	641	641
q13	8196	3216	3211	3211
q14	301	274	269	269
q15	588	545	523	523
q16	571	495	503	495
q17	1945	1784	1721	1721
q18	8691	8366	8247	8247
q19	1642	1602	1610	1602
q20	2248	2018	1993	1993
q21	5775	5553	5248	5248
q22	550	481	472	472
Total cold run time: 67092 ms
Total hot run time: 59306 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.63% (8615/23518)
Line Coverage: 28.68% (70010/244084)
Region Coverage: 27.67% (36240/130950)
Branch Coverage: 24.37% (18514/75968)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c10c1d1da590c6ddbe25719e54e8ac4368fec265_c10c1d1da590c6ddbe25719e54e8ac4368fec265/report/index.html

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit c10c1d1da590c6ddbe25719e54e8ac4368fec265, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	951	363	339	339
query2	6414	2109	1943	1943
query3	6666	209	200	200
query4	27161	22516	22399	22399
query5	5850	577	517	517
query6	266	187	185	185
query7	4577	282	263	263
query8	244	206	208	206
query9	8221	2589	2686	2589
query10	456	251	253	251
query11	16191	15506	15545	15506
query12	135	79	77	77
query13	1636	347	326	326
query14	11851	7090	7078	7078
query15	238	179	192	179
query16	6455	286	261	261
query17	1880	509	500	500
query18	1932	273	268	268
query19	285	140	133	133
query20	80	82	81	81
query21	185	98	92	92
query22	5084	4571	4452	4452
query23	31824	31192	31211	31192
query24	10959	2802	2824	2802
query25	589	359	351	351
query26	1537	141	149	141
query27	2873	283	280	280
query28	7134	1945	1934	1934
query29	1527	410	390	390
query30	284	141	145	141
query31	953	778	758	758
query32	89	58	58	58
query33	718	278	275	275
query34	846	447	451	447
query35	871	789	770	770
query36	1313	1199	1307	1199
query37	106	74	74	74
query38	3374	3312	3277	3277
query39	1314	1293	1260	1260
query40	188	92	92	92
query41	38	42	36	36
query42	93	90	93	90
query43	558	517	492	492
query44	1052	699	715	699
query45	194	185	173	173
query46	1071	639	676	639
query47	1669	1546	1576	1546
query48	345	261	260	260
query49	1046	340	330	330
query50	732	367	326	326
query51	5415	5184	5361	5184
query52	94	88	88	88
query53	215	150	156	150
query54	1414	584	569	569
query55	98	93	89	89
query56	217	202	206	202
query57	983	972	975	972
query58	231	203	204	203
query59	2835	2708	2652	2652
query60	272	248	249	248
query61	91	89	93	89
query62	642	464	456	456
query63	174	145	147	145
query64	4967	1727	1689	1689
query65	3331	3268	3260	3260
query66	1125	338	336	336
query67	15823	15014	14941	14941
query68	13149	536	534	534
query69	521	253	254	253
query70	1794	1555	1525	1525
query71	492	234	236	234
query72	5749	3565	3529	3529
query73	3081	315	331	315
query74	6969	6396	6508	6396
query75	5473	2291	2353	2291
query76	6334	1164	1176	1164
query77	687	287	268	268
query78	9075	8940	8649	8649
query79	1073	525	505	505
query80	553	390	383	383
query81	455	213	214	213
query82	208	107	100	100
query83	185	137	137	137
query84	249	54	53	53
query85	946	280	283	280
query86	393	376	393	376
query87	3581	3365	3358	3358
query88	2910	2258	2261	2258
query89	332	266	256	256
query90	1859	212	191	191
query91	119	91	100	91
query92	69	55	52	52
query93	1386	491	484	484
query94	835	190	189	189
query95	464	418	406	406
query96	623	314	313	313
query97	4273	4213	4179	4179
query98	206	197	187	187
query99	1064	846	842	842
Total cold run time: 293089 ms
Total hot run time: 179161 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit c10c1d1da590c6ddbe25719e54e8ac4368fec265, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5514	5128	5091	5091
q2	407	167	158	158
q3	1443	1151	1157	1151
q4	1074	888	849	849
q5	3145	3097	3058	3058
q6	220	140	127	127
q7	961	524	539	524
q8	2137	2276	2219	2219
q9	6651	6657	6626	6626
q10	3183	3078	3114	3078
q11	341	225	220	220
q12	386	236	234	234
q13	4400	3638	3621	3621
q14	249	219	209	209
q15	617	546	550	546
q16	461	399	395	395
q17	1039	574	515	515
q18	7100	6789	6821	6789
q19	1647	1496	1590	1496
q20	571	335	330	330
q21	2881	2440	2476	2440
q22	379	330	323	323
Total cold run time: 44806 ms
Total hot run time: 39999 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5085	5011	5039	5011
q2	345	248	241	241
q3	3360	3310	3303	3303
q4	2147	2017	1980	1980
q5	5904	5919	5905	5905
q6	224	124	124	124
q7	2367	1940	1923	1923
q8	3541	3663	3670	3663
q9	9061	9012	8977	8977
q10	3867	3890	3909	3890
q11	583	475	493	475
q12	812	619	651	619
q13	3843	3203	3176	3176
q14	320	281	260	260
q15	601	551	547	547
q16	556	535	489	489
q17	2042	1804	1795	1795
q18	8683	8279	8397	8279
q19	1731	1602	1690	1602
q20	2299	1992	1966	1966
q21	5733	5342	5358	5342
q22	539	539	510	510
Total cold run time: 63643 ms
Total hot run time: 60077 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.97 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 27.8 seconds inserted 10000000 Rows, about 359K ops/s
storage size: 17183861102 Bytes

@ByteYue ByteYue force-pushed the add_http_to_control_be_log_level branch from c10c1d1 to d03bca0 Compare January 4, 2024 12:07
@ByteYue
Copy link
Contributor Author

ByteYue commented Jan 4, 2024

run buildall

Copy link
Contributor

github-actions bot commented Jan 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit d03bca0534e953607daf558c6296bb1c54e69ce2, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5499	5122	5166	5122
q2	389	166	198	166
q3	1481	1176	1139	1139
q4	1089	876	822	822
q5	3184	3100	3036	3036
q6	221	133	134	133
q7	998	559	530	530
q8	2150	2232	2275	2232
q9	6696	6658	6650	6650
q10	3194	3139	3187	3139
q11	337	227	226	226
q12	386	239	229	229
q13	4383	3641	3619	3619
q14	247	212	213	212
q15	579	534	528	528
q16	470	410	414	410
q17	1043	534	533	533
q18	7098	6713	6714	6713
q19	1637	1544	1481	1481
q20	551	322	322	322
q21	2897	2491	2514	2491
q22	399	334	347	334
Total cold run time: 44928 ms
Total hot run time: 40067 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5109	5053	5082	5053
q2	349	235	226	226
q3	3348	3342	3304	3304
q4	2146	2039	2057	2039
q5	5932	5918	5902	5902
q6	222	125	126	125
q7	2397	1907	1920	1907
q8	3552	3653	3676	3653
q9	8949	8975	9002	8975
q10	3851	3892	3936	3892
q11	582	479	491	479
q12	798	625	650	625
q13	3873	3199	3214	3199
q14	308	265	268	265
q15	600	531	522	522
q16	551	481	494	481
q17	2045	1795	1813	1795
q18	8755	8313	8341	8313
q19	1737	1690	1682	1682
q20	2255	1994	1963	1963
q21	5703	5264	5352	5264
q22	574	499	478	478
Total cold run time: 63636 ms
Total hot run time: 60142 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit d03bca0534e953607daf558c6296bb1c54e69ce2, data reload: false

------ Round 1 ----------------------------------
q1	17746	5081	5060	5060
q2	2029	159	140	140
q3	10620	1090	1153	1090
q4	10186	732	777	732
q5	7797	2870	2857	2857
q6	210	132	132	132
q7	941	524	493	493
q8	9235	1969	1989	1969
q9	6808	6404	6364	6364
q10	8233	3022	3051	3022
q11	430	209	216	209
q12	389	229	221	221
q13	17976	3592	3626	3592
q14	239	225	210	210
q15	571	522	516	516
q16	450	401	399	399
q17	956	429	465	429
q18	7199	6636	6640	6636
q19	1585	1360	1370	1360
q20	705	351	339	339
q21	2802	2321	2355	2321
q22	383	311	331	311
Total cold run time: 107490 ms
Total hot run time: 38402 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5161	5118	5171	5118
q2	334	246	247	246
q3	3277	3269	3242	3242
q4	2106	2023	2011	2011
q5	5762	5766	5728	5728
q6	217	123	123	123
q7	2299	1899	1909	1899
q8	3369	3410	3446	3410
q9	8803	8733	8689	8689
q10	3775	3828	3844	3828
q11	563	482	470	470
q12	801	620	644	620
q13	7055	3220	3189	3189
q14	288	258	268	258
q15	571	513	514	513
q16	547	521	490	490
q17	1909	1750	1773	1750
q18	8660	8308	8269	8269
q19	1608	1581	1565	1565
q20	2177	1938	1957	1938
q21	5529	5307	5222	5222
q22	534	491	455	455
Total cold run time: 65345 ms
Total hot run time: 59033 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.64% (8618/23522)
Line Coverage: 28.67% (70015/244214)
Region Coverage: 27.64% (36234/131079)
Branch Coverage: 24.34% (18506/76040)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d03bca0534e953607daf558c6296bb1c54e69ce2_d03bca0534e953607daf558c6296bb1c54e69ce2/report/index.html

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit d03bca0534e953607daf558c6296bb1c54e69ce2, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	923	342	329	329
query2	6432	1969	2031	1969
query3	6655	211	205	205
query4	27760	22416	22430	22416
query5	4008	501	513	501
query6	270	182	189	182
query7	4573	267	262	262
query8	243	199	192	192
query9	8248	2506	2462	2462
query10	545	251	257	251
query11	16478	15641	15648	15641
query12	131	75	77	75
query13	1617	314	311	311
query14	11442	7098	7006	7006
query15	216	181	189	181
query16	6383	271	265	265
query17	1875	505	485	485
query18	1937	260	263	260
query19	185	139	128	128
query20	79	71	76	71
query21	179	95	95	95
query22	5079	4898	4712	4712
query23	31819	31271	31066	31066
query24	12116	2845	2849	2845
query25	611	339	330	330
query26	1747	143	138	138
query27	2930	267	270	267
query28	6996	1854	1854	1854
query29	2104	379	376	376
query30	289	145	148	145
query31	989	759	775	759
query32	83	62	55	55
query33	736	267	267	267
query34	876	441	434	434
query35	861	783	783	783
query36	1319	1174	1233	1174
query37	185	62	66	62
query38	3343	3299	3291	3291
query39	1334	1274	1272	1272
query40	306	91	86	86
query41	39	36	34	34
query42	87	79	86	79
query43	511	502	489	489
query44	1090	689	699	689
query45	191	188	181	181
query46	1074	636	632	632
query47	1661	1540	1541	1540
query48	328	252	251	251
query49	1197	318	308	308
query50	794	361	332	332
query51	5333	5263	5259	5259
query52	88	88	78	78
query53	215	142	143	142
query54	1360	560	557	557
query55	99	84	81	81
query56	195	182	187	182
query57	1068	957	916	916
query58	235	197	202	197
query59	2774	2715	2604	2604
query60	224	231	234	231
query61	88	87	86	86
query62	679	443	446	443
query63	170	150	142	142
query64	5911	1771	1711	1711
query65	3307	3241	3257	3241
query66	1388	346	323	323
query67	15751	15528	15380	15380
query68	11395	518	536	518
query69	503	242	250	242
query70	1698	1545	1428	1428
query71	495	224	220	220
query72	5727	3676	3581	3581
query73	2374	306	300	300
query74	6996	6484	6538	6484
query75	4784	2329	2253	2253
query76	6281	1091	1092	1091
query77	650	254	250	250
query78	9101	8632	8774	8632
query79	1038	501	489	489
query80	533	343	341	341
query81	455	212	210	210
query82	201	92	94	92
query83	164	133	132	132
query84	241	53	53	53
query85	946	286	279	279
query86	408	381	404	381
query87	3536	3368	3364	3364
query88	3154	2234	2222	2222
query89	326	270	267	267
query90	1900	194	188	188
query91	121	90	95	90
query92	54	51	56	51
query93	1873	489	474	474
query94	812	177	174	174
query95	460	409	387	387
query96	632	317	318	317
query97	4258	4159	4173	4159
query98	209	196	195	195
query99	1121	843	831	831
Total cold run time: 292205 ms
Total hot run time: 179006 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.8 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17183697743 Bytes

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 5, 2024
Copy link
Contributor

github-actions bot commented Jan 5, 2024

PR approved by at least one committer and no changes requested.

@yiguolei yiguolei merged commit d61fec3 into apache:master Jan 6, 2024
27 of 28 checks passed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
BiteTheDDDDt pushed a commit to apache/doris-website that referenced this pull request Nov 4, 2024
code pr: apache/doris#28785

# Versions 

- [x] dev
- [x] 3.0
- [x] 2.1
- [ ] 2.0

# Languages

- [x] Chinese
- [x] English
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants