Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](nereids)mv selection should rule out shadow index from schema change #37154

Merged
merged 2 commits into from
Jul 4, 2024

Conversation

starocean999
Copy link
Contributor

schema change would introduce a shadow index. MV selection should rule out the shadow index.

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@starocean999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39864 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 666cba9c2c00c8ad520bd98448374d0d5ba42383, data reload: false

------ Round 1 ----------------------------------
q1	17934	4664	4413	4413
q2	2555	189	194	189
q3	11248	1172	1100	1100
q4	11306	840	875	840
q5	8192	2681	2652	2652
q6	223	140	142	140
q7	961	587	602	587
q8	9228	2083	2061	2061
q9	8849	6471	6515	6471
q10	8903	3684	3744	3684
q11	449	232	232	232
q12	395	238	219	219
q13	18910	2953	2923	2923
q14	258	226	220	220
q15	517	485	471	471
q16	482	378	369	369
q17	952	735	694	694
q18	8012	7380	7334	7334
q19	6794	1486	1443	1443
q20	672	320	327	320
q21	4779	3171	3939	3171
q22	388	331	333	331
Total cold run time: 122007 ms
Total hot run time: 39864 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4412	4253	4203	4203
q2	368	261	268	261
q3	2944	2745	2708	2708
q4	1865	1550	1616	1550
q5	5215	5242	5231	5231
q6	222	132	131	131
q7	2114	1702	1723	1702
q8	3164	3333	3308	3308
q9	8317	8298	8256	8256
q10	3811	3617	3642	3617
q11	569	499	478	478
q12	773	570	622	570
q13	16531	2981	2962	2962
q14	301	257	261	257
q15	520	483	489	483
q16	481	406	417	406
q17	1788	1470	1460	1460
q18	7626	7440	7330	7330
q19	1705	1694	1552	1552
q20	1981	1790	1767	1767
q21	4911	4684	4719	4684
q22	609	550	547	547
Total cold run time: 70227 ms
Total hot run time: 53463 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170712 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 666cba9c2c00c8ad520bd98448374d0d5ba42383, data reload: false

query1	920	387	374	374
query2	6483	2452	2461	2452
query3	6659	209	222	209
query4	21713	17494	17287	17287
query5	4188	474	476	474
query6	272	175	160	160
query7	4599	287	284	284
query8	317	310	296	296
query9	8560	2361	2345	2345
query10	611	315	283	283
query11	10774	10188	10092	10092
query12	136	85	83	83
query13	1636	366	360	360
query14	10008	7671	7506	7506
query15	243	192	192	192
query16	7887	282	275	275
query17	1870	559	539	539
query18	1980	288	280	280
query19	201	161	156	156
query20	94	83	86	83
query21	214	134	125	125
query22	4685	4219	4120	4120
query23	33934	32933	32894	32894
query24	12154	2800	2874	2800
query25	696	396	381	381
query26	1813	154	157	154
query27	2981	314	309	309
query28	7301	2054	2040	2040
query29	1195	622	602	602
query30	291	154	152	152
query31	975	734	742	734
query32	87	51	54	51
query33	779	294	295	294
query34	947	472	468	468
query35	732	613	596	596
query36	1105	926	932	926
query37	286	76	75	75
query38	2919	2744	2765	2744
query39	858	801	798	798
query40	277	122	122	122
query41	56	53	52	52
query42	112	95	101	95
query43	600	558	558	558
query44	1176	718	708	708
query45	190	165	159	159
query46	1079	718	700	700
query47	1878	1778	1769	1769
query48	368	287	287	287
query49	1180	411	413	411
query50	774	379	379	379
query51	6880	6742	6778	6742
query52	107	93	96	93
query53	362	290	288	288
query54	895	435	455	435
query55	77	72	72	72
query56	276	255	282	255
query57	1182	1064	1095	1064
query58	245	239	232	232
query59	3644	3252	3085	3085
query60	315	290	268	268
query61	93	92	93	92
query62	655	447	442	442
query63	322	282	285	282
query64	9930	2223	1719	1719
query65	3159	3094	3091	3091
query66	1378	328	328	328
query67	15412	14971	14950	14950
query68	4765	558	548	548
query69	656	387	308	308
query70	1184	1151	1162	1151
query71	438	271	262	262
query72	8015	5249	2750	2750
query73	751	323	323	323
query74	6005	5564	5528	5528
query75	3980	2657	2674	2657
query76	3046	973	943	943
query77	638	317	305	305
query78	10478	9743	9796	9743
query79	2310	516	508	508
query80	1453	459	465	459
query81	571	214	220	214
query82	967	102	106	102
query83	322	169	168	168
query84	272	83	87	83
query85	2104	276	261	261
query86	477	320	319	319
query87	3270	3102	3075	3075
query88	4051	2340	2338	2338
query89	468	375	386	375
query90	1964	194	190	190
query91	126	98	99	98
query92	65	50	50	50
query93	2433	495	489	489
query94	1406	187	186	186
query95	405	313	316	313
query96	589	264	272	264
query97	3172	3003	3009	3003
query98	229	214	201	201
query99	1177	827	837	827
Total cold run time: 283512 ms
Total hot run time: 170712 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 666cba9c2c00c8ad520bd98448374d0d5ba42383, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.67	0.10	0.11
query5	0.53	0.48	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.06	0.05	0.04
query9	0.56	0.49	0.50
query10	0.54	0.56	0.56
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.59	0.58	0.58
query14	0.76	0.78	0.79
query15	0.84	0.81	0.79
query16	0.36	0.36	0.35
query17	0.99	1.02	0.98
query18	0.23	0.25	0.22
query19	1.76	1.67	1.76
query20	0.01	0.01	0.01
query21	15.40	0.83	0.66
query22	4.39	7.59	1.68
query23	18.29	1.27	1.23
query24	2.19	0.22	0.23
query25	0.14	0.08	0.09
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.23	1.02	0.99
query29	12.63	3.32	3.22
query30	0.25	0.06	0.06
query31	2.87	0.38	0.38
query32	3.30	0.46	0.46
query33	2.90	2.89	2.92
query34	16.97	4.40	4.34
query35	4.51	4.42	4.48
query36	0.64	0.49	0.46
query37	0.18	0.15	0.16
query38	0.16	0.14	0.15
query39	0.05	0.03	0.04
query40	0.17	0.14	0.16
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.03	0.05
Total cold run time: 109.51 s
Total hot run time: 30.07 s

@starocean999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39609 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 666cba9c2c00c8ad520bd98448374d0d5ba42383, data reload: false

------ Round 1 ----------------------------------
q1	17602	4414	4295	4295
q2	2015	188	183	183
q3	10490	1166	1122	1122
q4	10170	766	769	766
q5	7460	2622	2626	2622
q6	217	134	136	134
q7	942	586	592	586
q8	9221	2060	2032	2032
q9	8666	6448	6442	6442
q10	9002	3741	3668	3668
q11	439	230	233	230
q12	498	242	229	229
q13	17772	2992	2967	2967
q14	278	215	218	215
q15	529	491	490	490
q16	517	387	373	373
q17	956	626	700	626
q18	7836	7409	7337	7337
q19	4821	1509	1544	1509
q20	656	329	318	318
q21	4996	3132	3226	3132
q22	382	333	338	333
Total cold run time: 115465 ms
Total hot run time: 39609 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4380	4275	4312	4275
q2	377	266	270	266
q3	2984	2730	2876	2730
q4	1968	1729	1702	1702
q5	5554	5551	5450	5450
q6	235	140	133	133
q7	2154	1835	1814	1814
q8	3239	3388	3360	3360
q9	8615	8601	8750	8601
q10	4092	3949	3993	3949
q11	579	488	497	488
q12	815	675	661	661
q13	17277	3137	3133	3133
q14	310	269	283	269
q15	528	474	488	474
q16	476	431	430	430
q17	1821	1515	1516	1515
q18	8048	7953	7880	7880
q19	1781	1656	1592	1592
q20	2163	1864	1873	1864
q21	4977	4981	4760	4760
q22	611	564	556	556
Total cold run time: 72984 ms
Total hot run time: 55902 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172330 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 666cba9c2c00c8ad520bd98448374d0d5ba42383, data reload: false

query1	901	376	370	370
query2	6463	2331	2370	2331
query3	6632	200	205	200
query4	19733	17472	17344	17344
query5	3664	479	479	479
query6	246	192	157	157
query7	4593	297	285	285
query8	316	293	287	287
query9	8567	2370	2359	2359
query10	583	297	275	275
query11	10446	9904	9848	9848
query12	115	90	81	81
query13	1657	384	371	371
query14	9495	7696	6251	6251
query15	228	186	180	180
query16	7734	277	266	266
query17	1903	566	541	541
query18	1963	276	278	276
query19	198	157	157	157
query20	91	80	83	80
query21	213	128	130	128
query22	4389	3928	4029	3928
query23	33785	33535	33778	33535
query24	10624	2810	2811	2810
query25	583	386	372	372
query26	724	163	161	161
query27	2253	327	319	319
query28	5761	2139	2135	2135
query29	882	633	615	615
query30	252	162	155	155
query31	983	773	776	773
query32	94	50	57	50
query33	663	300	282	282
query34	859	497	468	468
query35	739	628	651	628
query36	1137	964	972	964
query37	145	79	86	79
query38	2961	2838	2871	2838
query39	909	836	816	816
query40	214	123	125	123
query41	52	55	54	54
query42	107	100	99	99
query43	593	536	526	526
query44	1062	734	759	734
query45	195	157	158	157
query46	1064	702	691	691
query47	1843	1766	1751	1751
query48	389	301	294	294
query49	852	403	410	403
query50	763	379	385	379
query51	6842	6771	6665	6665
query52	101	118	90	90
query53	362	299	291	291
query54	848	457	459	457
query55	73	74	74	74
query56	277	275	256	256
query57	1115	1080	1053	1053
query58	240	232	245	232
query59	3335	3215	3045	3045
query60	303	272	274	272
query61	93	92	95	92
query62	619	459	439	439
query63	309	290	282	282
query64	8455	2287	1749	1749
query65	3142	3110	3111	3110
query66	737	322	322	322
query67	15550	15109	14748	14748
query68	4617	546	535	535
query69	640	402	317	317
query70	1073	1096	1061	1061
query71	398	286	274	274
query72	8793	5714	5747	5714
query73	747	326	326	326
query74	5870	5456	5432	5432
query75	3438	2649	2656	2649
query76	2562	896	920	896
query77	656	304	294	294
query78	10313	9828	9662	9662
query79	2133	517	514	514
query80	2254	456	457	456
query81	589	221	221	221
query82	1051	107	101	101
query83	292	174	171	171
query84	259	85	87	85
query85	1104	272	264	264
query86	441	317	338	317
query87	3276	3063	3068	3063
query88	3122	2430	2418	2418
query89	469	378	369	369
query90	1785	187	182	182
query91	125	100	100	100
query92	59	49	53	49
query93	1900	509	504	504
query94	1215	189	190	189
query95	401	307	317	307
query96	590	271	265	265
query97	3174	2978	3037	2978
query98	221	199	191	191
query99	1220	856	844	844
Total cold run time: 267806 ms
Total hot run time: 172330 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 666cba9c2c00c8ad520bd98448374d0d5ba42383, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.23	0.06	0.05
query4	1.68	0.09	0.10
query5	0.49	0.48	0.48
query6	1.14	0.72	0.72
query7	0.03	0.02	0.01
query8	0.06	0.05	0.05
query9	0.54	0.49	0.50
query10	0.55	0.55	0.53
query11	0.15	0.12	0.12
query12	0.15	0.13	0.12
query13	0.59	0.58	0.58
query14	0.77	0.79	0.78
query15	0.85	0.82	0.81
query16	0.38	0.36	0.37
query17	1.02	1.04	1.05
query18	0.21	0.23	0.27
query19	1.77	1.70	1.81
query20	0.01	0.01	0.01
query21	15.44	0.77	0.67
query22	4.48	7.47	1.37
query23	18.34	1.39	1.30
query24	2.16	0.21	0.22
query25	0.15	0.08	0.10
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.19	1.02	1.01
query29	12.65	3.23	3.20
query30	0.26	0.06	0.05
query31	2.87	0.37	0.38
query32	3.29	0.47	0.46
query33	2.89	2.91	2.92
query34	17.14	4.41	4.40
query35	4.48	4.46	4.52
query36	0.64	0.46	0.48
query37	0.18	0.16	0.16
query38	0.16	0.14	0.15
query39	0.04	0.03	0.03
query40	0.17	0.16	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.8 s
Total hot run time: 30.05 s

@starocean999 starocean999 marked this pull request as ready for review July 3, 2024 06:34
@@ -145,7 +145,8 @@ private List<SyncMaterializationContext> createSyncMvContexts(OlapTable olapTabl
keyCount += column.isKey() ? 1 : 0;
}
for (Map.Entry<String, Long> entry : olapTable.getIndexNameToId().entrySet()) {
if (entry.getValue() != baseIndexId) {
long indexId = entry.getValue();
if (indexId != baseIndexId && !olapTable.isShadowIndex(indexId)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add some comment to explain shadowIndex

@starocean999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40563 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit afb00eee5e6c0679316a0e04ab050896008cd7b3, data reload: false

------ Round 1 ----------------------------------
q1	17605	4263	4256	4256
q2	2014	193	185	185
q3	10453	1210	1142	1142
q4	10189	738	772	738
q5	7565	2645	2631	2631
q6	219	137	137	137
q7	945	587	592	587
q8	9225	2065	2046	2046
q9	8964	6472	6457	6457
q10	8891	3689	3672	3672
q11	468	240	232	232
q12	494	245	232	232
q13	17765	2999	2980	2980
q14	270	230	227	227
q15	521	472	478	472
q16	514	371	375	371
q17	964	663	638	638
q18	8061	7399	7407	7399
q19	7599	1513	1508	1508
q20	649	329	321	321
q21	4971	4024	3993	3993
q22	398	349	339	339
Total cold run time: 118744 ms
Total hot run time: 40563 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4400	4215	4229	4215
q2	382	268	275	268
q3	3043	2887	2920	2887
q4	1970	1723	1766	1723
q5	5479	5500	5418	5418
q6	223	136	128	128
q7	2239	1833	1826	1826
q8	3218	3429	3439	3429
q9	8693	8796	8676	8676
q10	4133	3666	3835	3666
q11	579	508	522	508
q12	808	661	636	636
q13	17110	3117	3176	3117
q14	300	271	284	271
q15	513	485	502	485
q16	483	444	423	423
q17	1791	1498	1482	1482
q18	8081	7847	7844	7844
q19	1790	1753	1512	1512
q20	2123	1867	1847	1847
q21	5079	4799	4835	4799
q22	622	534	541	534
Total cold run time: 73059 ms
Total hot run time: 55694 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171283 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit afb00eee5e6c0679316a0e04ab050896008cd7b3, data reload: false

query1	935	373	362	362
query2	6433	2474	2279	2279
query3	6626	201	223	201
query4	19125	17619	17175	17175
query5	3767	463	466	463
query6	258	165	168	165
query7	4585	300	297	297
query8	343	304	296	296
query9	8523	2392	2363	2363
query10	585	324	294	294
query11	10480	9949	9923	9923
query12	115	91	83	83
query13	1653	377	383	377
query14	10254	6887	7697	6887
query15	239	194	188	188
query16	7746	273	281	273
query17	1885	556	546	546
query18	1878	288	281	281
query19	197	154	163	154
query20	92	81	83	81
query21	218	160	131	131
query22	4255	4034	3934	3934
query23	33634	33639	33635	33635
query24	11030	2892	2872	2872
query25	608	410	404	404
query26	735	160	160	160
query27	2246	327	343	327
query28	6239	2169	2139	2139
query29	937	659	683	659
query30	265	165	168	165
query31	997	767	762	762
query32	98	59	58	58
query33	762	316	313	313
query34	918	481	493	481
query35	776	656	629	629
query36	1180	987	977	977
query37	138	79	88	79
query38	2956	2891	2795	2795
query39	892	841	804	804
query40	211	129	128	128
query41	58	59	58	58
query42	120	104	102	102
query43	570	552	548	548
query44	1245	765	744	744
query45	282	163	166	163
query46	1078	730	697	697
query47	1842	1775	1762	1762
query48	391	311	304	304
query49	856	421	446	421
query50	774	389	388	388
query51	6989	6636	6703	6636
query52	106	91	96	91
query53	355	284	299	284
query54	904	458	450	450
query55	74	73	80	73
query56	289	257	267	257
query57	1120	1046	1038	1038
query58	259	262	246	246
query59	3338	3219	3506	3219
query60	312	278	279	278
query61	96	92	92	92
query62	600	437	442	437
query63	314	298	296	296
query64	8933	2395	1877	1877
query65	3149	3098	3090	3090
query66	769	338	338	338
query67	15603	15127	15005	15005
query68	7844	572	555	555
query69	704	446	362	362
query70	1186	1062	1071	1062
query71	518	289	281	281
query72	8647	5344	2790	2790
query73	803	327	326	326
query74	5885	5536	5562	5536
query75	5191	2620	2718	2620
query76	4523	973	972	972
query77	790	299	306	299
query78	10333	9818	9693	9693
query79	8618	523	524	523
query80	1108	474	473	473
query81	546	219	221	219
query82	749	107	107	107
query83	354	175	171	171
query84	273	83	85	83
query85	1316	291	281	281
query86	432	334	304	304
query87	3309	3122	3092	3092
query88	4510	2467	2466	2466
query89	548	401	405	401
query90	1952	191	189	189
query91	132	102	100	100
query92	66	51	49	49
query93	6587	531	513	513
query94	1211	191	194	191
query95	418	320	323	320
query96	615	272	270	270
query97	3229	3005	2987	2987
query98	216	201	196	196
query99	1191	818	852	818
Total cold run time: 289044 ms
Total hot run time: 171283 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.9 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit afb00eee5e6c0679316a0e04ab050896008cd7b3, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.10	0.09
query5	0.49	0.49	0.49
query6	1.14	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.55	0.49	0.48
query10	0.52	0.54	0.54
query11	0.15	0.12	0.11
query12	0.15	0.13	0.12
query13	0.60	0.58	0.58
query14	0.80	0.79	0.77
query15	0.84	0.81	0.82
query16	0.36	0.34	0.38
query17	1.00	0.97	1.05
query18	0.24	0.25	0.23
query19	1.81	1.66	1.66
query20	0.02	0.01	0.01
query21	15.93	0.79	0.65
query22	4.49	6.96	2.42
query23	18.28	1.34	1.30
query24	2.15	0.22	0.23
query25	0.17	0.09	0.07
query26	0.26	0.17	0.18
query27	0.08	0.08	0.08
query28	13.25	1.02	0.99
query29	12.57	3.25	3.25
query30	0.26	0.06	0.06
query31	2.86	0.38	0.39
query32	3.28	0.50	0.47
query33	2.93	2.84	2.95
query34	17.12	4.46	4.39
query35	4.45	4.48	4.45
query36	0.66	0.47	0.46
query37	0.18	0.15	0.16
query38	0.15	0.15	0.16
query39	0.04	0.04	0.03
query40	0.17	0.14	0.15
query41	0.09	0.05	0.06
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.22 s
Total hot run time: 30.9 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 3, 2024
Copy link
Contributor

github-actions bot commented Jul 3, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Jul 3, 2024

PR approved by anyone and no changes requested.

@starocean999 starocean999 merged commit 841e39a into apache:master Jul 4, 2024
26 of 29 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 16, 2024
…hange (apache#37154)

schema change would introduce a shadow index. MV selection should rule
out the shadow index.
dataroaring pushed a commit that referenced this pull request Jul 16, 2024
…ntext #37701 (#37872)

## Proposed changes

cherry-pick 3.0
pr: #37701
commit: e478570


pr: #37154
commit: 841e39a

---------

Co-authored-by: starocean999 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged not-merge/2.1 reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants