Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](mtmv) Support querying rewrite by materialized view when insert and insert overwrite dml #38115

Merged
merged 1 commit into from
Jul 22, 2024

Conversation

seawinde
Copy link
Contributor

@seawinde seawinde commented Jul 19, 2024

Proposed changes

Support querying rewrite by materialized view when DML such as insert and insert overwrite , into outfile etc.
this is default enable, you can disable it by set enable_dml_materialized_view_rewrite = false
and the enable_materialized_view_rewrite is only control DQL.

It would not rewrite by materialized view when query sql use external table.
If you want rewrite by materialized view when use external table in DML,
you should set enable_dml_materialized_view_rewrite_when_base_table_unawareness = true;
this is default false.

the pr is the replacement for #37561

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@seawinde
Copy link
Contributor Author

run buildall

@seawinde
Copy link
Contributor Author

run buildall

Copy link
Contributor

@zfr9527 zfr9527 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 19, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39676 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit feb0d60795b843b8180dc2fe86f6358ea0a7630e, data reload: false

------ Round 1 ----------------------------------
q1	17633	4754	4266	4266
q2	2008	191	187	187
q3	10457	1270	1094	1094
q4	10190	801	750	750
q5	7526	2704	2681	2681
q6	224	134	136	134
q7	952	591	607	591
q8	9214	2090	2053	2053
q9	8795	6584	6556	6556
q10	8764	3769	3846	3769
q11	512	244	249	244
q12	444	231	231	231
q13	17762	2971	2981	2971
q14	283	239	240	239
q15	540	486	473	473
q16	488	383	373	373
q17	966	681	708	681
q18	8030	7533	7334	7334
q19	6508	1291	1478	1291
q20	683	338	346	338
q21	4987	3141	3239	3141
q22	353	288	279	279
Total cold run time: 117319 ms
Total hot run time: 39676 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4449	4371	4238	4238
q2	375	254	267	254
q3	2991	2928	2940	2928
q4	2022	1641	1784	1641
q5	5594	5560	5479	5479
q6	228	132	129	129
q7	2236	1882	1839	1839
q8	3234	3396	3394	3394
q9	8794	8781	8881	8781
q10	4124	3887	3790	3790
q11	590	506	498	498
q12	827	630	659	630
q13	16034	3205	3190	3190
q14	312	285	279	279
q15	528	498	504	498
q16	502	432	443	432
q17	1844	1518	1508	1508
q18	8166	7849	7911	7849
q19	1715	1529	1655	1529
q20	2198	1883	1854	1854
q21	5083	4915	4712	4712
q22	625	496	493	493
Total cold run time: 72471 ms
Total hot run time: 55945 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174071 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit feb0d60795b843b8180dc2fe86f6358ea0a7630e, data reload: false

query1	928	379	368	368
query2	6451	1808	1833	1808
query3	6664	207	215	207
query4	23566	17438	17378	17378
query5	3727	480	475	475
query6	289	165	160	160
query7	4590	288	283	283
query8	238	187	185	185
query9	8448	2451	2454	2451
query10	436	274	262	262
query11	10812	10169	10101	10101
query12	110	79	83	79
query13	1633	368	363	363
query14	10403	7566	7670	7566
query15	214	165	160	160
query16	7576	343	323	323
query17	1326	578	541	541
query18	1809	273	268	268
query19	199	160	147	147
query20	89	80	85	80
query21	203	132	130	130
query22	4313	4137	4004	4004
query23	33820	33900	33765	33765
query24	10956	2941	2954	2941
query25	609	383	385	383
query26	699	146	144	144
query27	2258	274	276	274
query28	5873	2103	2106	2103
query29	861	637	634	634
query30	264	159	153	153
query31	953	796	748	748
query32	97	53	52	52
query33	647	295	292	292
query34	893	484	503	484
query35	705	587	569	569
query36	1134	979	965	965
query37	155	81	91	81
query38	2984	2961	2838	2838
query39	910	868	868	868
query40	236	122	122	122
query41	47	44	46	44
query42	113	98	94	94
query43	502	458	465	458
query44	1109	756	736	736
query45	210	161	158	158
query46	1092	746	749	746
query47	1868	1816	1782	1782
query48	359	295	296	295
query49	819	405	407	405
query50	783	385	390	385
query51	6887	6815	6701	6701
query52	107	91	94	91
query53	357	286	290	286
query54	845	450	440	440
query55	78	74	71	71
query56	278	256	287	256
query57	1130	1065	1046	1046
query58	240	250	250	250
query59	2808	2584	2576	2576
query60	333	277	270	270
query61	93	93	92	92
query62	812	631	636	631
query63	316	286	284	284
query64	9165	2227	1656	1656
query65	3200	3139	3119	3119
query66	756	343	347	343
query67	15555	14966	15040	14966
query68	4573	545	540	540
query69	668	446	408	408
query70	1206	1135	1128	1128
query71	402	288	273	273
query72	7426	5614	6013	5614
query73	771	327	326	326
query74	6158	5667	5730	5667
query75	3397	2698	2735	2698
query76	2639	909	1005	909
query77	624	299	310	299
query78	9647	10518	9164	9164
query79	3460	520	526	520
query80	949	482	467	467
query81	568	217	220	217
query82	747	142	132	132
query83	190	166	167	166
query84	273	88	85	85
query85	885	305	291	291
query86	452	289	303	289
query87	3281	3095	3208	3095
query88	4123	2368	2389	2368
query89	476	377	384	377
query90	1740	191	189	189
query91	196	102	97	97
query92	60	47	49	47
query93	3622	517	509	509
query94	1014	216	210	210
query95	414	323	321	321
query96	607	272	279	272
query97	3237	3108	3022	3022
query98	223	199	197	197
query99	1566	1251	1238	1238
Total cold run time: 273534 ms
Total hot run time: 174071 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.39 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit feb0d60795b843b8180dc2fe86f6358ea0a7630e, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.67	0.07	0.07
query5	0.50	0.48	0.49
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.54	0.49	0.48
query10	0.54	0.55	0.54
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.59	0.59	0.59
query14	0.76	0.79	0.76
query15	0.85	0.82	0.80
query16	0.34	0.35	0.35
query17	0.99	1.04	1.03
query18	0.23	0.21	0.22
query19	1.85	1.84	1.75
query20	0.01	0.01	0.01
query21	15.39	0.78	0.68
query22	4.58	7.23	1.70
query23	18.24	1.36	1.24
query24	2.16	0.24	0.22
query25	0.15	0.08	0.09
query26	0.29	0.20	0.21
query27	0.45	0.23	0.23
query28	13.17	1.01	1.00
query29	12.63	3.42	3.34
query30	0.25	0.05	0.05
query31	2.87	0.40	0.38
query32	3.28	0.46	0.47
query33	2.92	2.92	2.90
query34	17.03	4.41	4.33
query35	4.48	4.38	4.41
query36	0.65	0.46	0.48
query37	0.18	0.15	0.16
query38	0.15	0.15	0.14
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.09	0.04	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.95 s
Total hot run time: 30.39 s

@morrySnow morrySnow merged commit afb2dcc into apache:master Jul 22, 2024
32 of 33 checks passed
dataroaring pushed a commit that referenced this pull request Jul 24, 2024
…sert and insert overwrite dml (#38115)

Support querying rewrite by materialized view when DML such as insert
and insert overwrite , into outfile etc.
this is default enable, you can disable it by `set
enable_dml_materialized_view_rewrite = false`
and the `enable_materialized_view_rewrite` is only control DQL.

It would not rewrite by materialized view when query sql use external
table.
If you want rewrite by materialized view when use external table in DML,
you should `set
enable_dml_materialized_view_rewrite_when_base_table_unawareness =
true;`
this is default false.
seawinde added a commit to seawinde/doris that referenced this pull request Aug 6, 2024
…sert and insert overwrite dml (apache#38115)

Support querying rewrite by materialized view when DML such as insert
and insert overwrite , into outfile etc.
this is default enable, you can disable it by `set
enable_dml_materialized_view_rewrite = false`
and the `enable_materialized_view_rewrite` is only control DQL.

It would not rewrite by materialized view when query sql use external
table.
If you want rewrite by materialized view when use external table in DML,
you should `set
enable_dml_materialized_view_rewrite_when_base_table_unawareness =
true;`
this is default false.
yiguolei pushed a commit that referenced this pull request Aug 7, 2024
## Proposed changes

pr: #38115
commitId: 2b29288

pr: #38008
commitId: c6b924d

pr: #37929
commitId: d44fcdc
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants