Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bugfix](hive)Prevent multiple fs from being generated for 2.0 (#36954) #38045

Merged
merged 1 commit into from
Jul 23, 2024

Conversation

wuwenchi
Copy link
Contributor

Proposed changes

bp: #36954

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49864 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 347e24090fd4695257fae959e272ce1367f89416, data reload: false

------ Round 1 ----------------------------------
q1	18005	4342	4349	4342
q2	2098	161	150	150
q3	10452	1908	1912	1908
q4	10329	1276	1305	1276
q5	8421	3858	3893	3858
q6	240	125	124	124
q7	2048	1607	1578	1578
q8	9257	2703	2696	2696
q9	10580	10214	10198	10198
q10	8647	3543	3547	3543
q11	418	233	252	233
q12	464	308	301	301
q13	18340	3986	4008	3986
q14	357	327	324	324
q15	501	453	470	453
q16	693	572	573	572
q17	1116	977	984	977
q18	7264	7032	6912	6912
q19	1750	1674	1643	1643
q20	557	316	319	316
q21	4426	4040	4074	4040
q22	547	461	434	434
Total cold run time: 116510 ms
Total hot run time: 49864 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4281	4286	4278	4278
q2	324	225	223	223
q3	4183	4167	4144	4144
q4	2748	2738	2761	2738
q5	7213	7011	7105	7011
q6	237	119	121	119
q7	3258	2844	2776	2776
q8	4372	4451	4482	4451
q9	16893	16719	16733	16719
q10	4205	4236	4278	4236
q11	780	693	660	660
q12	1034	861	851	851
q13	6806	3739	3701	3701
q14	457	423	418	418
q15	502	460	444	444
q16	741	683	689	683
q17	3877	3867	3842	3842
q18	8867	8746	8760	8746
q19	1718	1712	1650	1650
q20	2340	2123	2092	2092
q21	8390	8430	8480	8430
q22	1084	975	1038	975
Total cold run time: 84310 ms
Total hot run time: 79187 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203640 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 347e24090fd4695257fae959e272ce1367f89416, data reload: false

query1	932	386	410	386
query2	6549	2810	2605	2605
query3	6919	213	227	213
query4	20026	18032	17962	17962
query5	19748	6504	6469	6469
query6	289	214	229	214
query7	4152	318	316	316
query8	411	426	397	397
query9	3069	2665	2563	2563
query10	406	309	299	299
query11	11290	10842	10710	10710
query12	120	78	77	77
query13	5610	722	686	686
query14	17666	13714	13564	13564
query15	404	248	251	248
query16	6449	287	264	264
query17	1740	1455	873	873
query18	2316	415	407	407
query19	212	154	149	149
query20	82	78	82	78
query21	191	94	97	94
query22	5362	5166	5130	5130
query23	32407	31750	31931	31750
query24	6986	6445	6481	6445
query25	524	432	427	427
query26	529	174	164	164
query27	1845	303	297	297
query28	6143	2359	2310	2310
query29	2953	2878	2753	2753
query30	254	165	169	165
query31	918	741	747	741
query32	62	61	55	55
query33	414	265	265	265
query34	854	474	493	474
query35	1108	908	920	908
query36	1411	1311	1317	1311
query37	92	60	61	60
query38	3079	2954	2913	2913
query39	1385	1326	1306	1306
query40	205	98	90	90
query41	47	44	43	43
query42	91	87	89	87
query43	726	746	732	732
query44	1103	717	729	717
query45	253	235	239	235
query46	1222	955	991	955
query47	1886	1665	1874	1665
query48	1016	708	710	708
query49	619	393	367	367
query50	861	602	582	582
query51	4756	4678	4722	4678
query52	91	77	82	77
query53	448	334	323	323
query54	2629	2409	2482	2409
query55	84	83	84	83
query56	241	236	213	213
query57	1233	1080	1056	1056
query58	226	195	196	195
query59	4097	4059	3710	3710
query60	221	200	209	200
query61	97	100	97	97
query62	778	498	463	463
query63	486	343	346	343
query64	2467	1532	1392	1392
query65	3657	3570	3569	3569
query66	788	375	370	370
query67	16218	15541	17538	15541
query68	8748	649	647	647
query69	565	350	361	350
query70	1662	1509	1274	1274
query71	416	311	317	311
query72	6578	3507	3500	3500
query73	729	325	324	324
query74	6264	5871	5889	5871
query75	5102	3694	3710	3694
query76	5174	1125	1195	1125
query77	815	253	261	253
query78	12627	11678	11720	11678
query79	7024	626	623	623
query80	1049	410	399	399
query81	487	234	238	234
query82	1485	98	98	98
query83	166	135	136	135
query84	255	74	72	72
query85	910	334	328	328
query86	330	336	325	325
query87	3205	3043	2999	2999
query88	4720	2326	2320	2320
query89	342	276	285	276
query90	1936	205	213	205
query91	174	145	155	145
query92	58	50	53	50
query93	3400	586	574	574
query94	742	211	215	211
query95	1098	1055	1064	1055
query96	650	336	324	324
query97	6538	6284	6361	6284
query98	198	175	170	170
query99	2983	838	858	838
Total cold run time: 307883 ms
Total hot run time: 203640 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.34 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 347e24090fd4695257fae959e272ce1367f89416, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.04	0.04
query4	1.79	0.08	0.07
query5	0.54	0.52	0.52
query6	1.24	0.70	0.60
query7	0.02	0.01	0.00
query8	0.03	0.02	0.03
query9	0.50	0.48	0.49
query10	0.54	0.54	0.53
query11	0.11	0.08	0.08
query12	0.11	0.09	0.10
query13	0.61	0.61	0.61
query14	0.80	0.79	0.78
query15	0.78	0.77	0.76
query16	0.37	0.39	0.37
query17	1.00	1.03	0.98
query18	0.23	0.23	0.26
query19	1.90	1.86	1.88
query20	0.02	0.01	0.01
query21	15.53	0.57	0.54
query22	1.90	2.75	1.49
query23	17.25	1.08	0.96
query24	5.06	0.87	1.28
query25	0.45	0.07	0.05
query26	0.54	0.15	0.17
query27	0.04	0.04	0.03
query28	7.95	0.70	0.70
query29	12.62	2.29	2.16
query30	0.57	0.52	0.54
query31	2.82	0.40	0.37
query32	3.36	0.49	0.49
query33	3.08	3.05	3.06
query34	15.24	4.78	4.77
query35	4.86	4.86	4.86
query36	1.06	1.01	1.02
query37	0.06	0.05	0.04
query38	0.04	0.03	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 103.65 s
Total hot run time: 30.34 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 347e24090fd4695257fae959e272ce1367f89416 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.8 seconds inserted 10000000 Rows, about 458K ops/s

@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49909 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 347e24090fd4695257fae959e272ce1367f89416, data reload: false

------ Round 1 ----------------------------------
q1	17724	4384	4376	4376
q2	2091	159	154	154
q3	10418	1889	1929	1889
q4	10358	1252	1306	1252
q5	8421	3925	3879	3879
q6	233	129	149	129
q7	2044	1623	1602	1602
q8	9324	2727	2717	2717
q9	10710	10413	10272	10272
q10	8699	3518	3508	3508
q11	430	247	254	247
q12	479	306	313	306
q13	18358	3990	4014	3990
q14	355	326	329	326
q15	501	444	458	444
q16	695	584	600	584
q17	1142	979	960	960
q18	7335	6943	6789	6789
q19	1778	1656	1645	1645
q20	520	302	298	298
q21	4439	4146	4090	4090
q22	532	453	452	452
Total cold run time: 116586 ms
Total hot run time: 49909 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4346	4319	4290	4290
q2	321	222	225	222
q3	4177	4140	4117	4117
q4	2772	2758	2761	2758
q5	7152	7087	7028	7028
q6	240	120	120	120
q7	3250	2879	2838	2838
q8	4382	4542	4512	4512
q9	16778	16793	16645	16645
q10	4220	4269	4240	4240
q11	757	710	680	680
q12	1035	873	887	873
q13	6819	3748	3690	3690
q14	465	432	423	423
q15	507	455	450	450
q16	737	668	673	668
q17	3830	3838	3843	3838
q18	8785	8644	8823	8644
q19	1731	1712	1652	1652
q20	2360	2164	2107	2107
q21	8529	8494	8539	8494
q22	1064	1013	964	964
Total cold run time: 84257 ms
Total hot run time: 79253 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 204136 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 347e24090fd4695257fae959e272ce1367f89416, data reload: false

query1	939	418	379	379
query2	6545	2910	2790	2790
query3	6921	218	204	204
query4	20065	18015	17980	17980
query5	19741	6558	6535	6535
query6	285	217	228	217
query7	4154	309	315	309
query8	442	401	410	401
query9	3135	2712	2653	2653
query10	434	298	302	298
query11	11389	10631	10656	10631
query12	126	74	76	74
query13	5618	689	704	689
query14	18268	13131	13646	13131
query15	358	250	247	247
query16	6455	292	260	260
query17	1742	1449	872	872
query18	2326	408	408	408
query19	209	155	158	155
query20	80	84	81	81
query21	194	96	101	96
query22	5275	5029	5010	5010
query23	32565	31834	31941	31834
query24	6895	6535	6489	6489
query25	523	437	418	418
query26	526	168	162	162
query27	1870	299	296	296
query28	6152	2383	2331	2331
query29	2802	2769	2705	2705
query30	242	171	172	171
query31	915	744	733	733
query32	69	65	63	63
query33	406	257	252	252
query34	842	481	482	481
query35	1127	943	959	943
query36	1277	1388	1258	1258
query37	93	60	63	60
query38	3071	2936	2919	2919
query39	1365	1344	1332	1332
query40	214	96	92	92
query41	46	45	45	45
query42	88	93	85	85
query43	742	695	689	689
query44	1151	719	736	719
query45	251	239	238	238
query46	1222	949	992	949
query47	1868	1705	1811	1705
query48	1014	720	709	709
query49	618	377	382	377
query50	861	637	629	629
query51	4806	4691	4750	4691
query52	98	76	84	76
query53	446	326	315	315
query54	2688	2501	2460	2460
query55	88	76	87	76
query56	242	229	226	226
query57	1144	1131	1168	1131
query58	221	200	201	200
query59	3996	4061	4198	4061
query60	214	208	226	208
query61	99	97	99	97
query62	804	455	489	455
query63	490	335	345	335
query64	2268	1537	1481	1481
query65	3622	3558	3617	3558
query66	786	386	376	376
query67	16065	17141	15410	15410
query68	8346	653	661	653
query69	571	349	354	349
query70	1624	1427	1438	1427
query71	420	315	330	315
query72	6520	3461	3471	3461
query73	731	320	312	312
query74	6218	5864	5853	5853
query75	4511	3660	3670	3660
query76	4621	1152	1142	1142
query77	560	264	262	262
query78	12502	11647	12219	11647
query79	8681	653	640	640
query80	2420	400	396	396
query81	507	236	232	232
query82	1160	98	96	96
query83	179	134	132	132
query84	260	72	71	71
query85	1400	335	336	335
query86	352	294	328	294
query87	3209	2985	3061	2985
query88	5036	2342	2366	2342
query89	380	294	291	291
query90	1762	202	211	202
query91	176	158	141	141
query92	60	54	55	54
query93	4837	566	564	564
query94	900	213	204	204
query95	1109	1082	1047	1047
query96	645	333	331	331
query97	6441	6418	6357	6357
query98	177	177	172	172
query99	3066	909	965	909
Total cold run time: 310974 ms
Total hot run time: 204136 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.56 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 347e24090fd4695257fae959e272ce1367f89416, data reload: false

query1	0.02	0.02	0.01
query2	0.07	0.03	0.03
query3	0.24	0.05	0.05
query4	1.80	0.06	0.06
query5	0.54	0.52	0.52
query6	1.25	0.62	0.61
query7	0.01	0.01	0.00
query8	0.04	0.02	0.02
query9	0.52	0.47	0.48
query10	0.54	0.52	0.53
query11	0.11	0.08	0.08
query12	0.11	0.10	0.09
query13	0.62	0.62	0.61
query14	0.79	0.79	0.80
query15	0.78	0.77	0.78
query16	0.39	0.36	0.35
query17	1.02	0.98	1.02
query18	0.22	0.24	0.25
query19	1.90	1.84	1.89
query20	0.02	0.01	0.01
query21	15.51	0.55	0.57
query22	1.94	1.79	1.54
query23	17.40	0.92	0.96
query24	5.50	1.92	0.86
query25	0.41	0.11	0.04
query26	0.62	0.15	0.15
query27	0.04	0.04	0.05
query28	6.90	0.76	0.77
query29	12.65	2.34	2.30
query30	0.55	0.59	0.54
query31	2.80	0.40	0.36
query32	3.39	0.49	0.49
query33	3.04	3.08	3.10
query34	15.27	4.80	4.79
query35	4.85	4.82	4.85
query36	1.06	1.00	1.01
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.3 s
Total hot run time: 30.56 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 347e24090fd4695257fae959e272ce1367f89416 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@morningman morningman merged commit a595705 into apache:branch-2.0 Jul 23, 2024
24 of 25 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants