Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](array-funcs)support arrays_overlap with invertedIndex #41161

Merged
merged 2 commits into from
Sep 25, 2024
Merged

[improve](array-funcs)support arrays_overlap with invertedIndex #41161

merged 2 commits into from
Sep 25, 2024

Conversation

amorynan
Copy link
Contributor

support arrays_overlap with invertedIndex

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@amorynan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41998 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit daf0fa72c535f1760547337dc27a5fec2af84d80, data reload: false

------ Round 1 ----------------------------------
q1	18129	7554	8060	7554
q2	2188	159	207	159
q3	11179	1163	1196	1163
q4	10673	807	731	731
q5	7943	3203	3141	3141
q6	240	158	155	155
q7	1029	637	635	635
q8	9504	2072	2040	2040
q9	6813	6353	6396	6353
q10	6999	2246	2250	2246
q11	427	251	254	251
q12	412	221	224	221
q13	17783	2948	3003	2948
q14	257	210	212	210
q15	584	515	497	497
q16	675	612	605	605
q17	985	855	831	831
q18	7225	6729	6769	6729
q19	1405	1008	990	990
q20	591	291	282	282
q21	4045	3266	3395	3266
q22	1100	1007	991	991
Total cold run time: 110186 ms
Total hot run time: 41998 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7313	7233	7369	7233
q2	330	224	222	222
q3	2894	2800	2848	2800
q4	1960	1694	1690	1690
q5	5344	5437	5420	5420
q6	228	147	144	144
q7	2099	1714	1698	1698
q8	3148	3351	3376	3351
q9	8321	8350	8350	8350
q10	3369	3314	3340	3314
q11	576	472	469	469
q12	801	586	601	586
q13	5970	3029	2982	2982
q14	285	256	253	253
q15	564	517	524	517
q16	715	663	676	663
q17	1752	1534	1541	1534
q18	7595	7445	7369	7369
q19	1676	1528	1539	1528
q20	2031	1776	1842	1776
q21	5280	5245	5157	5157
q22	1112	1053	972	972
Total cold run time: 63363 ms
Total hot run time: 58028 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190523 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit daf0fa72c535f1760547337dc27a5fec2af84d80, data reload: false

query1	931	374	362	362
query2	6514	2056	2004	2004
query3	6703	216	210	210
query4	34497	23443	23565	23443
query5	4365	472	447	447
query6	261	166	155	155
query7	4608	298	305	298
query8	299	227	229	227
query9	9954	2650	2627	2627
query10	438	295	274	274
query11	18216	15217	15109	15109
query12	141	96	96	96
query13	1668	423	402	402
query14	10572	7528	7376	7376
query15	286	169	174	169
query16	7780	469	459	459
query17	1702	572	552	552
query18	1969	312	311	311
query19	363	152	148	148
query20	118	109	111	109
query21	207	105	106	105
query22	4474	4388	4093	4093
query23	34431	33840	33936	33840
query24	11211	2917	2953	2917
query25	631	399	420	399
query26	1184	157	166	157
query27	2766	287	293	287
query28	8287	2441	2435	2435
query29	814	437	441	437
query30	325	161	165	161
query31	1010	777	811	777
query32	97	54	55	54
query33	779	313	303	303
query34	950	501	487	487
query35	876	729	716	716
query36	1104	925	908	908
query37	152	95	94	94
query38	4026	3946	3916	3916
query39	1495	1428	1411	1411
query40	216	96	99	96
query41	52	49	49	49
query42	119	101	94	94
query43	519	476	486	476
query44	1311	812	805	805
query45	199	165	171	165
query46	1134	790	780	780
query47	1881	1822	1839	1822
query48	454	375	361	361
query49	1160	424	421	421
query50	820	400	412	400
query51	7056	6922	6962	6922
query52	101	87	86	86
query53	255	177	180	177
query54	1198	466	492	466
query55	78	78	76	76
query56	266	274	262	262
query57	1204	1097	1105	1097
query58	259	234	253	234
query59	3085	2840	2897	2840
query60	313	282	285	282
query61	138	102	103	102
query62	836	676	651	651
query63	219	184	181	181
query64	4281	658	634	634
query65	3287	3185	3185	3185
query66	953	308	308	308
query67	15863	15665	15494	15494
query68	4936	566	562	562
query69	550	292	304	292
query70	1203	1127	1150	1127
query71	367	275	273	273
query72	7298	4115	3990	3990
query73	765	336	338	336
query74	9553	8941	8946	8941
query75	3418	2656	2654	2654
query76	3175	906	985	906
query77	559	292	295	292
query78	10028	9243	9133	9133
query79	2404	550	551	550
query80	1066	448	453	448
query81	573	244	243	243
query82	639	139	140	139
query83	249	135	140	135
query84	260	81	73	73
query85	1932	290	280	280
query86	489	288	301	288
query87	4439	4459	4232	4232
query88	3749	2342	2335	2335
query89	396	287	288	287
query90	2220	188	190	188
query91	177	146	160	146
query92	72	47	52	47
query93	1728	549	536	536
query94	1189	253	309	253
query95	361	250	256	250
query96	614	287	276	276
query97	3220	3113	3104	3104
query98	215	199	204	199
query99	1664	1326	1267	1267
Total cold run time: 302969 ms
Total hot run time: 190523 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.27% (9607/25775)
Line Coverage: 28.68% (79503/277246)
Region Coverage: 28.13% (41131/146220)
Branch Coverage: 24.77% (20965/84644)
Coverage Report: http://coverage.selectdb-in.cc/coverage/daf0fa72c535f1760547337dc27a5fec2af84d80_daf0fa72c535f1760547337dc27a5fec2af84d80/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 32.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit daf0fa72c535f1760547337dc27a5fec2af84d80, data reload: false

query1	0.04	0.05	0.04
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.52	0.52	0.49
query6	1.13	0.71	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.48
query10	0.55	0.56	0.55
query11	0.13	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	3.11	3.11	2.97
query15	0.89	0.82	0.81
query16	0.42	0.37	0.39
query17	1.00	1.04	1.07
query18	0.22	0.20	0.21
query19	1.99	1.85	2.02
query20	0.02	0.01	0.01
query21	15.35	0.60	0.58
query22	2.43	2.44	2.18
query23	17.31	0.83	0.73
query24	2.59	0.92	0.95
query25	0.24	0.07	0.04
query26	0.52	0.15	0.13
query27	0.04	0.04	0.03
query28	10.99	1.10	1.07
query29	12.58	3.28	3.22
query30	0.26	0.06	0.06
query31	2.88	0.37	0.37
query32	3.29	0.45	0.46
query33	2.92	3.06	2.99
query34	16.70	4.42	4.45
query35	4.47	4.41	4.46
query36	0.65	0.48	0.48
query37	0.08	0.05	0.06
query38	0.04	0.03	0.03
query39	0.04	0.02	0.02
query40	0.15	0.12	0.11
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.99 s
Total hot run time: 32.92 s

@amorynan
Copy link
Contributor Author

run p0

zzzxl1993
zzzxl1993 approved these changes Sep 24, 2024
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@amorynan
Copy link
Contributor Author

run buildall

airborne12
airborne12 approved these changes Sep 24, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 24, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 41207 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a229592fb3c95306a32d7fbdb0c6f8eeacf84b51, data reload: false

------ Round 1 ----------------------------------
q1	17602	7394	7311	7311
q2	2043	298	285	285
q3	12292	1103	1180	1103
q4	10592	751	822	751
q5	7753	2907	2867	2867
q6	235	150	145	145
q7	986	614	608	608
q8	9347	1986	1969	1969
q9	6559	6434	6439	6434
q10	6970	2274	2315	2274
q11	441	252	248	248
q12	407	216	214	214
q13	17819	2983	2983	2983
q14	252	209	214	209
q15	579	530	515	515
q16	673	631	609	609
q17	986	526	580	526
q18	7351	6841	6796	6796
q19	1408	1081	1053	1053
q20	585	287	297	287
q21	3923	3149	3048	3048
q22	1119	1001	972	972
Total cold run time: 109922 ms
Total hot run time: 41207 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7317	7267	7226	7226
q2	327	236	234	234
q3	3086	2958	2970	2958
q4	2097	1858	1861	1858
q5	5715	5694	5771	5694
q6	243	143	147	143
q7	2207	1827	1798	1798
q8	3387	3514	3513	3513
q9	8960	9132	8934	8934
q10	3534	3600	3583	3583
q11	592	503	489	489
q12	861	643	630	630
q13	9379	3149	3149	3149
q14	303	278	275	275
q15	586	526	552	526
q16	700	671	673	671
q17	1836	1603	1624	1603
q18	8345	7976	7957	7957
q19	1720	1407	1547	1407
q20	2162	1893	1913	1893
q21	5610	5191	5362	5191
q22	1117	1059	1073	1059
Total cold run time: 70084 ms
Total hot run time: 60791 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192246 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a229592fb3c95306a32d7fbdb0c6f8eeacf84b51, data reload: false

query1	975	382	394	382
query2	6298	2044	1951	1951
query3	8673	198	201	198
query4	33469	23831	23425	23425
query5	3524	469	453	453
query6	265	165	160	160
query7	4190	300	309	300
query8	282	221	216	216
query9	9630	2702	2713	2702
query10	472	295	263	263
query11	17809	15202	15332	15202
query12	152	97	98	97
query13	1538	428	415	415
query14	9584	7515	7301	7301
query15	245	172	183	172
query16	8105	460	473	460
query17	1683	604	585	585
query18	2176	311	316	311
query19	376	164	155	155
query20	123	112	116	112
query21	213	111	113	111
query22	4566	4489	4426	4426
query23	35445	34883	35172	34883
query24	11082	2922	2995	2922
query25	644	418	410	410
query26	1169	162	160	160
query27	2293	295	290	290
query28	7709	2483	2487	2483
query29	840	437	421	421
query30	260	157	162	157
query31	1019	814	795	795
query32	103	53	56	53
query33	767	309	297	297
query34	976	507	502	502
query35	879	737	716	716
query36	1091	934	943	934
query37	153	88	88	88
query38	4071	3873	3914	3873
query39	1510	1430	1427	1427
query40	209	98	101	98
query41	50	44	45	44
query42	113	96	94	94
query43	522	474	481	474
query44	1234	827	817	817
query45	195	164	164	164
query46	1157	708	707	707
query47	1934	1852	1902	1852
query48	450	358	374	358
query49	847	405	389	389
query50	862	406	408	406
query51	7041	6940	7056	6940
query52	97	88	87	87
query53	246	183	180	180
query54	1228	462	469	462
query55	78	75	77	75
query56	271	258	249	249
query57	1260	1128	1124	1124
query58	228	244	223	223
query59	3184	2879	2933	2879
query60	300	265	274	265
query61	104	103	101	101
query62	819	668	670	668
query63	216	185	188	185
query64	3966	629	636	629
query65	3289	3207	3167	3167
query66	696	305	331	305
query67	16014	15748	15659	15659
query68	4563	583	552	552
query69	611	289	306	289
query70	1159	1123	1095	1095
query71	396	275	279	275
query72	7489	3811	3994	3811
query73	773	342	354	342
query74	10496	9017	8969	8969
query75	3862	2676	2671	2671
query76	3484	930	883	883
query77	691	297	289	289
query78	10226	9344	9231	9231
query79	1709	621	611	611
query80	970	464	435	435
query81	550	239	240	239
query82	635	140	137	137
query83	341	140	132	132
query84	275	72	73	72
query85	1284	304	275	275
query86	377	291	304	291
query87	4661	4268	4349	4268
query88	3025	2426	2361	2361
query89	415	288	284	284
query90	2005	184	186	184
query91	181	145	147	145
query92	63	53	48	48
query93	1683	549	548	548
query94	839	304	310	304
query95	354	255	248	248
query96	613	272	278	272
query97	3304	3090	3100	3090
query98	216	211	194	194
query99	1583	1313	1292	1292
Total cold run time: 299802 ms
Total hot run time: 192246 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a229592fb3c95306a32d7fbdb0c6f8eeacf84b51, data reload: false

query1	0.05	0.05	0.04
query2	0.06	0.03	0.02
query3	0.24	0.07	0.07
query4	1.64	0.11	0.10
query5	0.51	0.50	0.51
query6	1.14	0.73	0.73
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.56	0.52	0.49
query10	0.55	0.56	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.59	0.59
query14	2.94	2.94	2.96
query15	0.89	0.83	0.83
query16	0.37	0.38	0.38
query17	1.02	1.02	1.02
query18	0.20	0.19	0.19
query19	1.98	1.85	1.97
query20	0.01	0.01	0.01
query21	15.35	0.59	0.58
query22	2.53	2.64	1.62
query23	17.33	0.80	0.87
query24	2.93	0.93	1.68
query25	0.26	0.07	0.10
query26	0.49	0.15	0.14
query27	0.05	0.04	0.04
query28	10.10	1.10	1.07
query29	12.62	3.20	3.18
query30	0.24	0.06	0.06
query31	2.89	0.39	0.38
query32	3.25	0.47	0.48
query33	3.03	3.05	3.09
query34	17.08	4.55	4.44
query35	4.49	4.49	4.54
query36	0.66	0.48	0.47
query37	0.09	0.05	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.15	0.12	0.12
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.87 s
Total hot run time: 32.66 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.28% (9607/25773)
Line Coverage: 28.70% (79537/277156)
Region Coverage: 28.14% (41135/146155)
Branch Coverage: 24.77% (20961/84606)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a229592fb3c95306a32d7fbdb0c6f8eeacf84b51_a229592fb3c95306a32d7fbdb0c6f8eeacf84b51/report/index.html

@airborne12 airborne12 merged commit b42a06c into apache:master Sep 25, 2024
24 of 28 checks passed
amorynan added a commit to amorynan/doris that referenced this pull request Sep 25, 2024
@amorynan
[improve](array-funcs)support arrays_overlap with invertedIndex (apac…
b12b3b5 
…he#41161)

support arrays_overlap with invertedIndex 

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
dataroaring pushed a commit that referenced this pull request Sep 29, 2024
@amorynan
[improve](array-funcs) accelerate arrays_overlap by inverted index (#…
d5e1026 
…41161) (#41286)

support arrays_overlap with invertedIndex
dataroaring added a commit that referenced this pull request Sep 30, 2024
@dataroaring
Revert "[improve](array-funcs) accelerate arrays_overlap by inverted …
1069a4d 
…index (#41161) (#41286)"

This reverts commit d5e1026.
@amorynan amorynan mentioned this pull request Oct 9, 2024
Merged
dataroaring pushed a commit that referenced this pull request Oct 10, 2024
@amorynan
[Improve](array-overlaps) improve array overlaps (#41571)
1fedfd5 
## Proposed changes
feature for array overlaps
backports: #41161 
#41495
Issue Number: close #xxx
amorynan added a commit to amorynan/doris that referenced this pull request Oct 18, 2024
@amorynan
[improve](array-funcs) accelerate arrays_overlap by inverted index (a…
585984f 
…pache#41161) (apache#41286)

support arrays_overlap with invertedIndex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

@airborne12 airborne12 airborne12 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.2-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@amorynan @doris-robot @airborne12 @zzzxl1993 @dataroaring