Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](nereids)mv selection should only consider visible indexes #38148

Merged
merged 2 commits into from
Jul 23, 2024

Conversation

starocean999
Copy link
Contributor

@starocean999 starocean999 commented Jul 19, 2024

the indexes in olap table are not always visible. Sometimes, schema change will introduce invisible index into olap table. And we should only use getVisibleIndexIdToMeta() to get visible indexes as candidates for mv selection.

@starocean999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

TPC-H: Total hot run time: 39895 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3d0ac8394b4432d6c14b36b646e4e21d2f113ced, data reload: false

------ Round 1 ----------------------------------
q1	17828	4450	4453	4450
q2	2625	197	192	192
q3	12105	1194	1008	1008
q4	11169	794	735	735
q5	7857	2713	2654	2654
q6	225	146	144	144
q7	966	611	615	611
q8	9506	2095	2115	2095
q9	8702	6545	6514	6514
q10	8688	3693	3790	3693
q11	471	251	254	251
q12	395	236	227	227
q13	17766	2996	2984	2984
q14	266	243	226	226
q15	521	496	499	496
q16	505	379	385	379
q17	968	652	752	652
q18	8009	7435	7422	7422
q19	7105	1458	1408	1408
q20	658	322	331	322
q21	4964	3155	3257	3155
q22	345	277	278	277
Total cold run time: 121644 ms
Total hot run time: 39895 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4405	4257	4206	4206
q2	379	271	292	271
q3	2961	2771	2727	2727
q4	1882	1628	1620	1620
q5	5293	5290	5272	5272
q6	222	131	133	131
q7	2117	1722	1733	1722
q8	3174	3374	3274	3274
q9	8380	8411	8358	8358
q10	3912	3710	3650	3650
q11	570	485	495	485
q12	788	609	613	609
q13	17254	3007	2972	2972
q14	301	274	271	271
q15	524	485	466	466
q16	468	412	428	412
q17	1760	1475	1486	1475
q18	7635	7465	7341	7341
q19	1669	1629	1452	1452
q20	2005	1773	1757	1757
q21	4913	4598	4548	4548
q22	580	477	515	477
Total cold run time: 71192 ms
Total hot run time: 53496 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171748 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3d0ac8394b4432d6c14b36b646e4e21d2f113ced, data reload: false

query1	912	375	367	367
query2	6482	1774	1792	1774
query3	6659	209	216	209
query4	28206	17451	17217	17217
query5	4230	505	497	497
query6	265	167	151	151
query7	4584	287	284	284
query8	252	194	195	194
query9	8458	2393	2376	2376
query10	448	276	280	276
query11	10512	9942	9983	9942
query12	138	89	86	86
query13	1647	376	364	364
query14	10282	6133	6984	6133
query15	218	162	169	162
query16	7638	392	474	392
query17	1277	561	550	550
query18	1815	287	285	285
query19	199	152	156	152
query20	89	83	82	82
query21	208	127	132	127
query22	4453	4041	4012	4012
query23	34404	32938	33145	32938
query24	12191	2860	2874	2860
query25	686	389	391	389
query26	1858	151	154	151
query27	2926	276	277	276
query28	7340	1988	1975	1975
query29	1113	644	633	633
query30	282	148	148	148
query31	1022	772	756	756
query32	98	56	57	56
query33	790	345	347	345
query34	917	478	477	477
query35	898	741	711	711
query36	1107	920	934	920
query37	271	81	81	81
query38	2825	2750	2769	2750
query39	856	816	794	794
query40	280	124	123	123
query41	49	49	48	48
query42	123	103	105	103
query43	508	465	457	457
query44	1225	725	722	722
query45	195	161	164	161
query46	1098	711	766	711
query47	1865	1784	1753	1753
query48	373	292	295	292
query49	1166	428	430	428
query50	786	389	392	389
query51	6936	6872	6818	6818
query52	105	94	94	94
query53	363	288	295	288
query54	893	457	451	451
query55	78	78	76	76
query56	305	282	293	282
query57	1180	1043	1022	1022
query58	264	244	252	244
query59	3035	2612	2585	2585
query60	310	275	294	275
query61	95	96	100	96
query62	822	659	660	659
query63	325	280	289	280
query64	10487	2230	7435	2230
query65	3147	3125	3136	3125
query66	1155	332	342	332
query67	15770	15069	15178	15069
query68	9181	551	569	551
query69	760	443	375	375
query70	1356	1074	1156	1074
query71	522	285	287	285
query72	9050	5468	6360	5468
query73	1695	338	323	323
query74	6199	5660	5631	5631
query75	5076	2689	2717	2689
query76	5427	1053	931	931
query77	805	309	324	309
query78	9781	8955	8958	8955
query79	8367	517	515	515
query80	1209	475	473	473
query81	579	221	234	221
query82	651	136	132	132
query83	320	161	164	161
query84	271	86	88	86
query85	1125	312	306	306
query86	361	310	302	302
query87	3294	3093	3171	3093
query88	4766	2362	2344	2344
query89	546	385	381	381
query90	2224	194	189	189
query91	128	101	100	100
query92	58	48	51	48
query93	5991	503	512	503
query94	1553	286	285	285
query95	397	315	306	306
query96	626	266	268	266
query97	3185	3053	2989	2989
query98	214	200	191	191
query99	1601	1262	1261	1261
Total cold run time: 308812 ms
Total hot run time: 171748 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3d0ac8394b4432d6c14b36b646e4e21d2f113ced, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.08
query5	0.49	0.48	0.48
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.56	0.49	0.49
query10	0.53	0.55	0.54
query11	0.14	0.11	0.12
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	0.77	0.78	0.78
query15	0.85	0.81	0.81
query16	0.38	0.36	0.37
query17	1.04	1.04	1.03
query18	0.22	0.22	0.22
query19	1.91	1.80	1.67
query20	0.02	0.01	0.01
query21	15.40	0.73	0.66
query22	4.54	6.94	2.14
query23	18.29	1.41	1.32
query24	2.15	0.24	0.23
query25	0.16	0.09	0.08
query26	0.29	0.21	0.21
query27	0.45	0.23	0.24
query28	13.23	1.02	0.99
query29	12.60	3.29	3.27
query30	0.27	0.07	0.06
query31	2.87	0.38	0.38
query32	3.28	0.48	0.48
query33	2.89	2.89	2.92
query34	17.04	4.36	4.36
query35	4.52	4.40	4.44
query36	0.65	0.48	0.47
query37	0.19	0.16	0.16
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.13	0.13
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.2 s
Total hot run time: 30.94 s

@starocean999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40011 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 09e9c4d84ba1b30bf15c897e1f84eb88195668f4, data reload: false

------ Round 1 ----------------------------------
q1	17614	4397	4280	4280
q2	2019	192	201	192
q3	10438	1206	1166	1166
q4	10179	766	768	766
q5	7539	2710	2782	2710
q6	224	140	139	139
q7	967	598	599	598
q8	9221	2037	2078	2037
q9	8725	6569	6501	6501
q10	8688	3733	3761	3733
q11	461	236	245	236
q12	514	224	233	224
q13	17770	2987	3012	2987
q14	276	235	242	235
q15	537	481	492	481
q16	497	382	386	382
q17	953	705	686	686
q18	7956	7512	7461	7461
q19	4625	1439	1383	1383
q20	672	329	331	329
q21	4943	3201	3246	3201
q22	351	284	287	284
Total cold run time: 115169 ms
Total hot run time: 40011 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4416	4248	4291	4248
q2	378	276	268	268
q3	2995	2709	2873	2709
q4	2022	1756	1671	1671
q5	5638	5504	5492	5492
q6	240	137	131	131
q7	2159	1763	1909	1763
q8	3277	3457	3386	3386
q9	8803	8748	8933	8748
q10	4115	3938	3813	3813
q11	598	496	488	488
q12	808	631	627	627
q13	16909	3223	3199	3199
q14	302	293	300	293
q15	542	494	480	480
q16	508	422	426	422
q17	1778	1520	1508	1508
q18	8048	7944	7784	7784
q19	1766	1471	1604	1471
q20	2098	1921	1879	1879
q21	5152	5031	4906	4906
q22	608	511	531	511
Total cold run time: 73160 ms
Total hot run time: 55797 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174316 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 09e9c4d84ba1b30bf15c897e1f84eb88195668f4, data reload: false

query1	926	380	383	380
query2	6449	1853	1837	1837
query3	6633	214	222	214
query4	28659	17639	17209	17209
query5	3659	489	495	489
query6	280	183	180	180
query7	4582	299	290	290
query8	257	194	192	192
query9	8735	2451	2431	2431
query10	430	293	266	266
query11	10645	10213	10214	10213
query12	115	84	83	83
query13	1637	374	366	366
query14	10409	7760	7690	7690
query15	220	170	174	170
query16	7217	485	503	485
query17	1133	565	526	526
query18	1708	284	282	282
query19	202	183	161	161
query20	89	90	82	82
query21	206	125	125	125
query22	4286	3993	4000	3993
query23	33975	33915	33581	33581
query24	11408	2974	2971	2971
query25	628	401	399	399
query26	764	162	154	154
query27	2435	282	277	277
query28	6626	2110	2109	2109
query29	882	629	626	626
query30	249	158	155	155
query31	954	773	767	767
query32	96	58	78	58
query33	766	339	342	339
query34	909	503	522	503
query35	861	772	761	761
query36	1142	967	945	945
query37	143	81	83	81
query38	2905	2853	2825	2825
query39	887	808	822	808
query40	203	119	120	119
query41	47	43	42	42
query42	127	104	99	99
query43	501	472	463	463
query44	1278	735	737	735
query45	199	164	164	164
query46	1085	737	746	737
query47	1858	1763	1752	1752
query48	366	299	296	296
query49	853	429	439	429
query50	787	411	396	396
query51	6861	6684	6661	6661
query52	114	91	104	91
query53	361	293	290	290
query54	908	450	448	448
query55	73	73	76	73
query56	297	283	280	280
query57	1168	1038	1049	1038
query58	254	246	276	246
query59	2964	2607	2717	2607
query60	323	277	284	277
query61	100	96	103	96
query62	814	624	649	624
query63	321	285	293	285
query64	9359	2223	1732	1732
query65	3198	3122	3109	3109
query66	762	342	337	337
query67	15582	14920	15089	14920
query68	4480	536	537	536
query69	478	335	340	335
query70	1106	1118	1141	1118
query71	391	286	293	286
query72	6898	5658	5581	5581
query73	759	328	326	326
query74	6080	5636	5720	5636
query75	3403	2775	2717	2717
query76	2649	944	902	902
query77	438	319	316	316
query78	10456	9976	8882	8882
query79	3101	535	531	531
query80	2142	483	488	483
query81	591	225	221	221
query82	723	145	139	139
query83	293	173	171	171
query84	285	90	91	90
query85	2081	331	310	310
query86	491	318	328	318
query87	3280	3102	3116	3102
query88	3797	2375	2388	2375
query89	511	397	399	397
query90	1848	203	200	200
query91	132	103	172	103
query92	61	49	50	49
query93	4727	519	513	513
query94	1218	284	298	284
query95	414	332	326	326
query96	616	284	272	272
query97	3212	3008	2999	2999
query98	225	205	194	194
query99	1529	1246	1235	1235
Total cold run time: 283285 ms
Total hot run time: 174316 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.83 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 09e9c4d84ba1b30bf15c897e1f84eb88195668f4, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.03	0.04
query3	0.23	0.05	0.05
query4	1.66	0.10	0.09
query5	0.51	0.50	0.49
query6	1.12	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.51
query10	0.57	0.55	0.55
query11	0.15	0.11	0.12
query12	0.15	0.13	0.12
query13	0.61	0.59	0.59
query14	0.77	0.79	0.76
query15	0.87	0.81	0.82
query16	0.36	0.36	0.37
query17	1.01	1.02	0.96
query18	0.23	0.22	0.22
query19	1.82	1.71	1.78
query20	0.01	0.00	0.00
query21	15.40	0.77	0.65
query22	4.01	7.46	2.10
query23	18.36	1.43	1.29
query24	2.13	0.22	0.21
query25	0.15	0.09	0.08
query26	0.30	0.21	0.22
query27	0.45	0.24	0.22
query28	13.30	1.02	1.01
query29	12.58	3.28	3.25
query30	0.25	0.06	0.06
query31	2.86	0.39	0.39
query32	3.27	0.47	0.48
query33	2.96	2.95	2.90
query34	17.01	4.33	4.35
query35	4.44	4.46	4.46
query36	0.64	0.46	0.48
query37	0.20	0.16	0.16
query38	0.15	0.16	0.15
query39	0.04	0.04	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.67 s
Total hot run time: 30.83 s

@starocean999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40041 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 86a8ecb457d4355f9952ed6f3c4cfc206fc8d08a, data reload: false

------ Round 1 ----------------------------------
q1	18232	4462	4232	4232
q2	2018	192	187	187
q3	11074	1162	1118	1118
q4	10559	819	828	819
q5	7592	2728	2712	2712
q6	227	141	136	136
q7	977	615	623	615
q8	9653	2132	2086	2086
q9	9099	6716	6640	6640
q10	9044	3790	3748	3748
q11	464	240	242	240
q12	401	218	218	218
q13	17773	2991	2993	2991
q14	279	248	234	234
q15	532	482	488	482
q16	510	386	375	375
q17	967	736	682	682
q18	8127	7447	7468	7447
q19	4124	1351	1352	1351
q20	703	327	325	325
q21	4935	3109	4100	3109
q22	361	305	294	294
Total cold run time: 117651 ms
Total hot run time: 40041 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4373	4293	4229	4229
q2	366	267	256	256
q3	2989	2945	2966	2945
q4	2057	1678	1703	1678
q5	5496	5519	5562	5519
q6	231	135	135	135
q7	2220	1844	1840	1840
q8	3269	3410	3331	3331
q9	8756	8724	8685	8685
q10	4048	3766	3822	3766
q11	570	480	487	480
q12	824	665	650	650
q13	16352	3195	3206	3195
q14	324	286	298	286
q15	529	487	493	487
q16	491	443	436	436
q17	1814	1584	1486	1486
q18	8129	8041	7592	7592
q19	1715	1357	1609	1357
q20	2080	1856	1860	1856
q21	5022	4955	4902	4902
q22	605	517	525	517
Total cold run time: 72260 ms
Total hot run time: 55628 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173589 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 86a8ecb457d4355f9952ed6f3c4cfc206fc8d08a, data reload: false

query1	912	368	362	362
query2	6168	1911	1821	1821
query3	6666	204	218	204
query4	28096	17363	17168	17168
query5	3846	473	484	473
query6	263	172	187	172
query7	4581	282	285	282
query8	240	191	192	191
query9	8539	2447	2430	2430
query10	452	273	279	273
query11	11476	10004	10082	10004
query12	136	89	81	81
query13	1648	378	375	375
query14	10181	7904	7698	7698
query15	222	175	173	173
query16	7561	479	463	463
query17	1572	534	540	534
query18	1272	269	284	269
query19	191	153	153	153
query20	86	84	80	80
query21	203	131	120	120
query22	4365	4023	3994	3994
query23	34123	33596	33680	33596
query24	10997	2885	2915	2885
query25	600	383	392	383
query26	1461	152	155	152
query27	2726	278	280	278
query28	7143	2077	2092	2077
query29	929	622	653	622
query30	244	152	159	152
query31	965	743	770	743
query32	98	56	54	54
query33	770	336	314	314
query34	958	505	490	490
query35	938	752	750	750
query36	1121	967	983	967
query37	152	89	92	89
query38	2935	2842	2845	2842
query39	941	845	861	845
query40	278	125	117	117
query41	47	43	49	43
query42	108	96	108	96
query43	522	474	458	458
query44	1241	735	739	735
query45	198	157	157	157
query46	1087	746	736	736
query47	1879	1752	1786	1752
query48	377	294	291	291
query49	898	415	427	415
query50	781	400	401	400
query51	6784	6737	6751	6737
query52	106	88	97	88
query53	357	302	295	295
query54	995	455	449	449
query55	76	75	74	74
query56	290	268	267	267
query57	1153	1063	1028	1028
query58	262	250	259	250
query59	2869	2681	2833	2681
query60	292	273	270	270
query61	94	94	97	94
query62	790	632	634	632
query63	319	286	292	286
query64	10328	2217	1667	1667
query65	3196	3106	3094	3094
query66	1098	371	336	336
query67	15313	14919	14869	14869
query68	6198	543	547	543
query69	616	410	358	358
query70	1111	1139	1144	1139
query71	482	278	277	277
query72	7566	5387	5366	5366
query73	778	321	324	321
query74	6090	5694	5744	5694
query75	3852	2686	2699	2686
query76	3818	897	889	889
query77	621	315	315	315
query78	9744	8925	9594	8925
query79	1871	526	531	526
query80	1582	468	468	468
query81	584	221	221	221
query82	753	133	135	133
query83	249	169	167	167
query84	260	85	86	85
query85	1331	331	357	331
query86	346	317	286	286
query87	3274	3084	3101	3084
query88	4002	2356	2366	2356
query89	475	377	370	370
query90	1606	192	191	191
query91	127	99	101	99
query92	65	50	49	49
query93	2347	505	503	503
query94	640	275	292	275
query95	384	319	318	318
query96	601	274	267	267
query97	3235	3025	3064	3025
query98	223	200	193	193
query99	1517	1253	1294	1253
Total cold run time: 283863 ms
Total hot run time: 173589 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.42 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 86a8ecb457d4355f9952ed6f3c4cfc206fc8d08a, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.66	0.08	0.08
query5	0.50	0.49	0.48
query6	1.13	0.73	0.72
query7	0.01	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.48	0.49
query10	0.53	0.54	0.53
query11	0.15	0.12	0.11
query12	0.14	0.13	0.12
query13	0.60	0.60	0.58
query14	0.76	0.79	0.78
query15	0.86	0.82	0.82
query16	0.36	0.36	0.38
query17	0.98	0.96	1.00
query18	0.23	0.23	0.22
query19	1.79	1.66	1.77
query20	0.01	0.01	0.01
query21	15.39	0.74	0.65
query22	3.70	8.03	1.65
query23	18.27	1.37	1.32
query24	2.19	0.22	0.23
query25	0.16	0.10	0.08
query26	0.29	0.21	0.21
query27	0.45	0.24	0.24
query28	13.24	1.03	1.02
query29	12.66	3.35	3.33
query30	0.25	0.06	0.05
query31	2.86	0.41	0.39
query32	3.26	0.48	0.47
query33	2.96	2.91	2.92
query34	17.05	4.37	4.33
query35	4.51	4.40	4.40
query36	0.67	0.46	0.49
query37	0.18	0.16	0.16
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.12	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.3 s
Total hot run time: 30.42 s

@starocean999 starocean999 marked this pull request as ready for review July 23, 2024 02:25
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 23, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit af94b06 into apache:master Jul 23, 2024
28 of 30 checks passed
dataroaring pushed a commit that referenced this pull request Jul 24, 2024
)

the indexes in olap table are not always visible. Sometimes, schema
change will introduce invisible index into olap table. And we should
only use getVisibleIndexIdToMeta() to get visible indexes as candidates
for mv selection.
@morrySnow morrySnow added usercase Important user case type label dev/3.0.1-merged and removed dev/3.0.1-merged labels Aug 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged not-merge/2.1 reviewed usercase Important user case type label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants