Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement](inverted index) Remove the check for inverted index file exists #36929

Merged
merged 5 commits into from
Jun 28, 2024

Conversation

csun5285
Copy link
Contributor

@csun5285 csun5285 commented Jun 27, 2024

Proposed changes

#36945

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@csun5285
Copy link
Contributor Author

run buildall

1 similar comment
@csun5285
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 50464 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d3533cd3ae2caa180f2f656ce1d99f309d04153f, data reload: false

------ Round 1 ----------------------------------
q1	17528	4410	4347	4347
q2	2072	154	148	148
q3	10268	1956	1917	1917
q4	10103	1259	1335	1259
q5	8637	3952	3993	3952
q6	234	126	127	126
q7	2097	1600	1583	1583
q8	9288	2760	2741	2741
q9	11042	11276	10799	10799
q10	8683	3519	3528	3519
q11	421	236	251	236
q12	468	296	303	296
q13	18351	3975	4020	3975
q14	368	349	323	323
q15	500	464	455	455
q16	673	572	569	569
q17	1144	972	926	926
q18	7375	6775	6856	6775
q19	1814	1687	1657	1657
q20	535	318	292	292
q21	4472	4197	4129	4129
q22	545	440	446	440
Total cold run time: 116618 ms
Total hot run time: 50464 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4340	4325	4321	4321
q2	321	222	222	222
q3	4211	4185	4152	4152
q4	2772	2746	2762	2746
q5	7290	7171	7161	7161
q6	237	119	121	119
q7	3272	2837	2825	2825
q8	4379	4490	4528	4490
q9	17484	17273	17232	17232
q10	4246	4313	4286	4286
q11	769	673	699	673
q12	1042	872	884	872
q13	6963	3745	3775	3745
q14	478	432	432	432
q15	500	458	449	449
q16	755	690	690	690
q17	3833	3907	3760	3760
q18	8787	8825	8821	8821
q19	1732	1710	1653	1653
q20	2406	2128	2111	2111
q21	8457	8481	8414	8414
q22	1046	974	998	974
Total cold run time: 85320 ms
Total hot run time: 80148 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203651 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d3533cd3ae2caa180f2f656ce1d99f309d04153f, data reload: false

query1	917	426	385	385
query2	6533	2899	2577	2577
query3	6926	203	205	203
query4	19769	17994	17898	17898
query5	19745	6463	6494	6463
query6	283	220	240	220
query7	4161	302	311	302
query8	437	410	410	410
query9	3111	2713	2633	2633
query10	406	320	314	314
query11	11321	10694	10609	10609
query12	124	77	76	76
query13	5601	688	707	688
query14	17819	13236	13500	13236
query15	354	242	248	242
query16	6487	285	270	270
query17	1704	1473	891	891
query18	2297	426	403	403
query19	212	153	155	153
query20	81	80	80	80
query21	189	94	90	90
query22	5184	5023	4991	4991
query23	32520	32073	31933	31933
query24	7042	6508	6552	6508
query25	503	420	417	417
query26	647	163	167	163
query27	2052	303	303	303
query28	6212	2390	2312	2312
query29	3088	2742	2742	2742
query30	243	166	172	166
query31	924	725	761	725
query32	73	69	57	57
query33	410	261	258	258
query34	843	478	483	478
query35	1124	920	979	920
query36	1372	1241	1232	1232
query37	97	62	60	60
query38	3094	2954	2962	2954
query39	1372	1352	1324	1324
query40	299	97	99	97
query41	45	44	45	44
query42	86	80	83	80
query43	832	684	631	631
query44	1120	717	723	717
query45	247	249	244	244
query46	1235	961	948	948
query47	1768	1876	1939	1876
query48	1012	712	708	708
query49	625	372	380	372
query50	874	607	617	607
query51	4809	4634	4732	4634
query52	98	86	91	86
query53	447	329	327	327
query54	2678	2472	2465	2465
query55	83	91	81	81
query56	255	210	184	184
query57	1133	1098	1088	1088
query58	225	198	199	198
query59	4003	3880	4025	3880
query60	208	218	214	214
query61	100	94	97	94
query62	848	497	496	496
query63	492	344	346	344
query64	2755	1579	1524	1524
query65	3659	3615	3547	3547
query66	781	386	386	386
query67	15556	15099	15451	15099
query68	10589	658	664	658
query69	572	353	350	350
query70	2177	1419	1573	1419
query71	414	311	316	311
query72	6682	3481	3487	3481
query73	736	324	308	308
query74	6298	5812	5808	5808
query75	5509	3762	3773	3762
query76	6906	1146	1163	1146
query77	1185	259	257	257
query78	12907	11734	11863	11734
query79	7171	629	642	629
query80	1054	407	412	407
query81	486	236	242	236
query82	830	104	102	102
query83	166	136	141	136
query84	256	70	72	70
query85	876	320	326	320
query86	326	336	293	293
query87	3234	2980	3032	2980
query88	4498	2327	2332	2327
query89	390	316	284	284
query90	1960	214	219	214
query91	187	143	140	140
query92	60	52	53	52
query93	6529	604	577	577
query94	711	212	210	210
query95	1118	1069	1065	1065
query96	649	330	330	330
query97	6570	6393	6502	6393
query98	189	178	175	175
query99	2940	900	820	820
Total cold run time: 315725 ms
Total hot run time: 203651 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.89% (8114/21417)
Line Coverage: 29.55% (66432/224850)
Region Coverage: 29.01% (34236/118000)
Branch Coverage: 24.87% (17571/70650)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d3533cd3ae2caa180f2f656ce1d99f309d04153f_d3533cd3ae2caa180f2f656ce1d99f309d04153f/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 31.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d3533cd3ae2caa180f2f656ce1d99f309d04153f, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.03
query3	0.25	0.06	0.06
query4	1.76	0.11	0.08
query5	0.53	0.53	0.53
query6	1.24	0.61	0.63
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.48	0.47
query10	0.53	0.53	0.54
query11	0.12	0.09	0.09
query12	0.11	0.08	0.09
query13	0.62	0.61	0.61
query14	0.76	0.80	0.78
query15	0.77	0.76	0.76
query16	0.40	0.37	0.39
query17	1.01	1.04	0.99
query18	0.23	0.25	0.25
query19	1.93	1.86	1.86
query20	0.02	0.01	0.01
query21	15.46	0.54	0.55
query22	1.95	2.18	2.11
query23	17.24	1.14	1.07
query24	5.10	0.84	1.24
query25	0.39	0.13	0.04
query26	0.52	0.15	0.17
query27	0.03	0.04	0.03
query28	7.95	0.72	0.71
query29	12.77	2.27	2.23
query30	0.55	0.49	0.55
query31	2.82	0.40	0.38
query32	3.37	0.50	0.50
query33	3.08	3.06	3.06
query34	15.24	4.80	4.81
query35	4.90	4.86	4.84
query36	1.04	1.02	1.02
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.15
query41	0.06	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 103.73 s
Total hot run time: 31.21 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d3533cd3ae2caa180f2f656ce1d99f309d04153f with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

@csun5285
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.88% (8112/21417)
Line Coverage: 29.54% (66419/224850)
Region Coverage: 29.00% (34224/118000)
Branch Coverage: 24.86% (17562/70650)
Coverage Report: http://coverage.selectdb-in.cc/coverage/85bca9f261e382d6fb08e1e434090e50b511f8ed_85bca9f261e382d6fb08e1e434090e50b511f8ed/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 50469 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85bca9f261e382d6fb08e1e434090e50b511f8ed, data reload: false

------ Round 1 ----------------------------------
q1	17731	4412	4348	4348
q2	2073	156	149	149
q3	10450	1933	1926	1926
q4	10446	1277	1318	1277
q5	8631	3944	3951	3944
q6	260	140	131	131
q7	2085	1582	1585	1582
q8	9671	2757	2729	2729
q9	13406	10405	10257	10257
q10	8813	3556	3581	3556
q11	412	248	241	241
q12	468	309	303	303
q13	18531	4016	4065	4016
q14	356	331	315	315
q15	509	466	477	466
q16	677	587	582	582
q17	1131	957	900	900
q18	7455	7216	7394	7216
q19	3240	1586	1627	1586
q20	685	350	319	319
q21	4526	4235	4152	4152
q22	570	474	476	474
Total cold run time: 122126 ms
Total hot run time: 50469 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4464	4382	4386	4382
q2	324	242	228	228
q3	4204	4232	4179	4179
q4	2768	2770	2767	2767
q5	7124	7132	7130	7130
q6	239	119	124	119
q7	3203	2815	2872	2815
q8	4377	4483	4471	4471
q9	16974	16861	16840	16840
q10	4225	4316	4249	4249
q11	733	698	690	690
q12	1061	837	857	837
q13	6911	3760	3757	3757
q14	452	424	425	424
q15	509	464	464	464
q16	734	683	682	682
q17	3823	3884	3851	3851
q18	8870	8815	8751	8751
q19	1728	1663	1677	1663
q20	2404	2084	2134	2084
q21	8579	8510	8475	8475
q22	1093	951	994	951
Total cold run time: 84799 ms
Total hot run time: 79809 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203950 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85bca9f261e382d6fb08e1e434090e50b511f8ed, data reload: false

query1	930	421	376	376
query2	6566	2732	2818	2732
query3	6925	204	206	204
query4	20017	18064	17939	17939
query5	19737	6503	6505	6503
query6	303	220	220	220
query7	4153	303	311	303
query8	433	439	450	439
query9	3121	2653	2605	2605
query10	425	301	303	301
query11	11264	10728	10637	10637
query12	125	76	74	74
query13	5597	699	700	699
query14	17795	13859	13519	13519
query15	368	247	240	240
query16	6454	297	264	264
query17	1698	1464	876	876
query18	2296	409	405	405
query19	223	152	153	152
query20	81	82	81	81
query21	195	98	97	97
query22	5241	4906	4989	4906
query23	32584	32212	32118	32118
query24	7004	6577	6593	6577
query25	540	451	438	438
query26	522	162	158	158
query27	1834	298	296	296
query28	6129	2365	2307	2307
query29	2850	2606	2608	2606
query30	249	164	168	164
query31	894	744	760	744
query32	73	66	64	64
query33	396	282	258	258
query34	844	462	482	462
query35	1127	917	889	889
query36	1344	1114	1314	1114
query37	88	63	60	60
query38	3085	3004	2944	2944
query39	1388	1347	1329	1329
query40	211	91	97	91
query41	47	44	45	44
query42	77	83	76	76
query43	664	666	702	666
query44	1175	716	714	714
query45	248	237	236	236
query46	1234	977	976	976
query47	1825	1792	2014	1792
query48	1020	702	707	702
query49	621	390	382	382
query50	874	613	628	613
query51	4830	4682	4619	4619
query52	101	86	81	81
query53	453	317	328	317
query54	2643	2429	2488	2429
query55	97	73	83	73
query56	269	225	213	213
query57	1154	1010	1144	1010
query58	222	213	199	199
query59	4589	4293	3994	3994
query60	226	223	192	192
query61	98	93	93	93
query62	760	494	488	488
query63	502	349	344	344
query64	2501	1540	1401	1401
query65	3654	3547	3570	3547
query66	795	385	376	376
query67	15650	16980	15270	15270
query68	8832	646	688	646
query69	564	365	353	353
query70	1555	1372	1371	1371
query71	411	312	319	312
query72	6553	3469	3527	3469
query73	734	317	324	317
query74	6329	5904	5894	5894
query75	5382	3701	3679	3679
query76	5339	1148	1210	1148
query77	902	253	263	253
query78	12779	11695	12286	11695
query79	8127	660	658	658
query80	1196	401	410	401
query81	490	231	237	231
query82	1435	106	98	98
query83	192	135	133	133
query84	265	70	71	70
query85	890	315	314	314
query86	336	295	309	295
query87	3306	3034	3045	3034
query88	5093	2353	2359	2353
query89	368	302	289	289
query90	1927	218	217	217
query91	175	140	139	139
query92	61	55	55	55
query93	4658	565	531	531
query94	706	217	207	207
query95	1097	1050	1042	1042
query96	677	324	327	324
query97	6484	6340	6404	6340
query98	185	186	184	184
query99	2913	860	903	860
Total cold run time: 311399 ms
Total hot run time: 203950 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85bca9f261e382d6fb08e1e434090e50b511f8ed, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.04	0.04
query4	1.83	0.06	0.06
query5	0.54	0.53	0.52
query6	1.23	0.62	0.63
query7	0.02	0.01	0.00
query8	0.03	0.02	0.02
query9	0.54	0.51	0.49
query10	0.54	0.53	0.54
query11	0.13	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.62	0.61
query14	0.76	0.78	0.79
query15	0.78	0.76	0.76
query16	0.38	0.36	0.38
query17	1.00	1.04	0.97
query18	0.21	0.27	0.24
query19	1.96	1.83	1.81
query20	0.01	0.01	0.01
query21	15.47	0.56	0.55
query22	2.34	2.86	2.05
query23	17.08	1.05	1.12
query24	6.83	0.83	1.20
query25	0.37	0.08	0.06
query26	0.73	0.16	0.16
query27	0.05	0.04	0.04
query28	6.38	0.77	0.78
query29	12.61	2.26	2.26
query30	0.54	0.53	0.52
query31	2.80	0.39	0.37
query32	3.37	0.50	0.49
query33	3.08	3.12	3.06
query34	15.27	4.78	4.83
query35	4.84	4.85	4.83
query36	1.06	1.01	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 104.26 s
Total hot run time: 31.12 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 85bca9f261e382d6fb08e1e434090e50b511f8ed with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

@airborne12 airborne12 changed the title [fix](inverted index) Remove the check for inverted index file exists [Improvement](inverted index) Remove the check for inverted index file exists Jun 28, 2024
@airborne12 airborne12 merged commit 132df2d into apache:branch-2.0 Jun 28, 2024
24 of 26 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants