Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](nereids)using mv's derived stats #35721

Merged
merged 3 commits into from
Jun 12, 2024
Merged

[opt](nereids)using mv's derived stats #35721

merged 3 commits into from
Jun 12, 2024

Conversation

englefly
Copy link
Contributor

@englefly englefly commented May 31, 2024 •
edited
Loading

Proposed changes

for a given mv, we have 2 stats. one is generated by stats analyzer, which is accurate, another is derived stats, which is derived when optimizer runs the mv query.
If a query tree contains a mv, we will use the mv's derived stats to derive stats for the whole tree.

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

run buildall

@englefly
Copy link
Contributor Author

englefly commented Jun 4, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40955 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 47567a36c88212ae8c819514a58625a4653eb166, data reload: false

------ Round 1 ----------------------------------
q1	17626	4395	4227	4227
q2	2012	196	202	196
q3	10440	1235	1242	1235
q4	10199	850	873	850
q5	7483	2688	2660	2660
q6	226	137	138	137
q7	993	605	605	605
q8	9218	2102	2105	2102
q9	9358	6682	6718	6682
q10	9291	3891	3884	3884
q11	455	246	250	246
q12	427	230	233	230
q13	17216	3234	3246	3234
q14	278	229	218	218
q15	519	467	472	467
q16	479	384	383	383
q17	971	605	633	605
q18	8323	7774	7744	7744
q19	5905	1399	1374	1374
q20	670	332	323	323
q21	5088	4008	3220	3220
q22	425	340	333	333
Total cold run time: 117602 ms
Total hot run time: 40955 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4507	4446	4421	4421
q2	377	276	274	274
q3	3379	3007	2928	2928
q4	2042	1668	1717	1668
q5	5482	5501	5538	5501
q6	230	128	130	128
q7	2163	1804	1807	1804
q8	3168	3345	3326	3326
q9	8598	8535	8531	8531
q10	4074	3788	3855	3788
q11	580	509	483	483
q12	793	607	575	575
q13	16943	3105	3125	3105
q14	305	278	272	272
q15	509	490	487	487
q16	487	428	424	424
q17	1800	1476	1500	1476
q18	8057	7637	7302	7302
q19	1726	1555	1543	1543
q20	3020	1797	1754	1754
q21	4833	4583	4668	4583
q22	915	519	548	519
Total cold run time: 73988 ms
Total hot run time: 54892 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169979 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 47567a36c88212ae8c819514a58625a4653eb166, data reload: false

query1	904	382	371	371
query2	6435	2602	2220	2220
query3	6647	211	207	207
query4	19114	17256	17210	17210
query5	4108	447	445	445
query6	235	157	148	148
query7	4589	299	287	287
query8	298	260	279	260
query9	8721	2430	2431	2430
query10	441	295	274	274
query11	10496	9930	9998	9930
query12	131	92	89	89
query13	1650	371	375	371
query14	10142	7639	7476	7476
query15	240	185	181	181
query16	7851	267	265	265
query17	1367	519	513	513
query18	1932	271	261	261
query19	193	148	146	146
query20	95	81	82	81
query21	200	123	129	123
query22	4489	4096	4120	4096
query23	33518	33112	32886	32886
query24	11171	2884	2901	2884
query25	612	357	355	355
query26	1201	150	153	150
query27	3035	321	332	321
query28	7548	2050	2059	2050
query29	866	602	591	591
query30	284	148	145	145
query31	964	780	732	732
query32	94	54	53	53
query33	763	286	271	271
query34	968	480	477	477
query35	728	604	606	604
query36	1088	934	905	905
query37	152	70	72	70
query38	2866	2764	2731	2731
query39	887	788	765	765
query40	209	130	121	121
query41	52	50	49	49
query42	129	96	99	96
query43	572	545	538	538
query44	1203	727	741	727
query45	203	166	168	166
query46	1077	696	726	696
query47	1897	1777	1780	1777
query48	380	293	309	293
query49	1017	412	414	412
query50	781	385	394	385
query51	6963	6740	6720	6720
query52	109	94	97	94
query53	357	288	286	286
query54	909	465	439	439
query55	72	74	73	73
query56	277	252	256	252
query57	1144	1085	1058	1058
query58	259	248	224	224
query59	3478	3349	3351	3349
query60	290	270	270	270
query61	88	89	86	86
query62	620	446	452	446
query63	321	287	288	287
query64	9961	2187	1732	1732
query65	3148	3124	3102	3102
query66	1344	326	317	317
query67	15460	15045	14960	14960
query68	4502	536	523	523
query69	448	298	305	298
query70	1092	1107	1084	1084
query71	406	284	287	284
query72	7159	2744	2613	2613
query73	745	327	330	327
query74	6056	5439	5431	5431
query75	3347	2699	2608	2608
query76	2417	914	928	914
query77	425	291	290	290
query78	10277	9892	9682	9682
query79	1449	515	512	512
query80	1204	442	449	442
query81	586	215	220	215
query82	1087	98	99	98
query83	269	168	176	168
query84	260	84	85	84
query85	1300	273	304	273
query86	414	297	335	297
query87	3325	3097	3090	3090
query88	3068	2409	2422	2409
query89	472	385	386	385
query90	1790	189	187	187
query91	125	95	97	95
query92	59	50	51	50
query93	1103	499	505	499
query94	1189	190	187	187
query95	407	323	318	318
query96	575	267	267	267
query97	3155	2970	2971	2970
query98	244	225	216	216
query99	1098	846	834	834
Total cold run time: 269396 ms
Total hot run time: 169979 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.93 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 47567a36c88212ae8c819514a58625a4653eb166, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.66	0.09	0.09
query5	0.49	0.47	0.49
query6	1.12	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.55	0.48	0.49
query10	0.55	0.54	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.11
query13	0.59	0.60	0.60
query14	0.77	0.78	0.78
query15	0.82	0.81	0.80
query16	0.37	0.36	0.36
query17	1.02	0.94	1.04
query18	0.23	0.22	0.28
query19	1.78	1.73	1.78
query20	0.02	0.01	0.01
query21	15.46	0.71	0.68
query22	4.56	6.68	2.20
query23	18.33	1.31	1.32
query24	1.79	0.24	0.19
query25	0.15	0.08	0.08
query26	0.26	0.18	0.18
query27	0.09	0.07	0.07
query28	13.47	1.01	0.99
query29	13.31	3.33	3.29
query30	0.24	0.06	0.05
query31	2.86	0.39	0.38
query32	3.31	0.47	0.47
query33	2.90	2.86	2.91
query34	17.13	4.43	4.44
query35	4.50	4.58	4.63
query36	0.65	0.46	0.47
query37	0.19	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.16	0.14	0.14
query41	0.09	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 110.43 s
Total hot run time: 30.93 s

@englefly englefly marked this pull request as ready for review June 11, 2024 07:50
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 11, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@englefly
Copy link
Contributor Author

run buildall

@englefly
Copy link
Contributor Author

run p0

@englefly
Copy link
Contributor Author

run performance

@doris-robot
Copy link

TPC-H: Total hot run time: 39429 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a37e32bdda3bcadcc6248db46c1b3088e919294b, data reload: false

------ Round 1 ----------------------------------
q1	17613	4304	4239	4239
q2	2027	201	190	190
q3	10447	1070	1081	1070
q4	10200	745	786	745
q5	7472	2678	2688	2678
q6	214	134	135	134
q7	940	596	594	594
q8	9205	2054	2065	2054
q9	9094	6476	6463	6463
q10	8820	3697	3727	3697
q11	432	233	230	230
q12	448	223	229	223
q13	17760	2959	2972	2959
q14	278	224	223	223
q15	510	483	470	470
q16	517	375	368	368
q17	963	651	764	651
q18	7981	7438	7375	7375
q19	5649	1329	1417	1329
q20	660	307	312	307
q21	4837	3107	3181	3107
q22	374	323	326	323
Total cold run time: 116441 ms
Total hot run time: 39429 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4445	4224	4232	4224
q2	363	263	266	263
q3	2970	2740	2970	2740
q4	1969	1609	1729	1609
q5	5548	5583	5522	5522
q6	233	137	139	137
q7	2188	1866	1899	1866
q8	3306	3427	3420	3420
q9	8712	8723	8759	8723
q10	4058	3872	3778	3778
q11	591	485	484	484
q12	790	629	653	629
q13	16199	3145	3139	3139
q14	304	272	280	272
q15	520	484	470	470
q16	528	442	419	419
q17	1787	1494	1509	1494
q18	8049	8001	7795	7795
q19	1810	1671	1605	1605
q20	2090	1883	1898	1883
q21	5089	4766	4889	4766
q22	624	524	568	524
Total cold run time: 72173 ms
Total hot run time: 55762 ms

@morrySnow
Copy link
Contributor

add description

@doris-robot
Copy link

TPC-DS: Total hot run time: 172813 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a37e32bdda3bcadcc6248db46c1b3088e919294b, data reload: false

query1	922	379	373	373
query2	6454	2380	2170	2170
query3	6631	211	212	211
query4	19367	17203	17262	17203
query5	3614	478	476	476
query6	240	162	157	157
query7	4577	310	292	292
query8	322	284	277	277
query9	8515	2350	2328	2328
query10	570	318	289	289
query11	10443	9981	9972	9972
query12	123	86	82	82
query13	1641	370	360	360
query14	9960	6432	7738	6432
query15	228	186	186	186
query16	7507	268	260	260
query17	1485	537	510	510
query18	1923	268	279	268
query19	196	155	153	153
query20	93	84	83	83
query21	198	149	121	121
query22	4309	4031	3928	3928
query23	33892	33731	33687	33687
query24	11304	2922	2826	2826
query25	614	380	370	370
query26	1192	163	156	156
query27	2883	351	324	324
query28	6964	2087	2081	2081
query29	895	622	637	622
query30	233	151	156	151
query31	957	758	763	758
query32	96	57	63	57
query33	778	282	287	282
query34	1102	488	495	488
query35	734	643	629	629
query36	1145	971	978	971
query37	166	74	72	72
query38	2940	2851	2831	2831
query39	923	835	815	815
query40	222	134	129	129
query41	58	51	54	51
query42	115	110	106	106
query43	593	568	536	536
query44	1254	726	748	726
query45	204	173	169	169
query46	1075	721	718	718
query47	1835	1757	1770	1757
query48	381	304	307	304
query49	886	399	415	399
query50	784	394	394	394
query51	6781	6697	6747	6697
query52	103	91	93	91
query53	368	288	285	285
query54	844	464	444	444
query55	73	74	72	72
query56	273	251	270	251
query57	1141	1037	1032	1032
query58	278	246	257	246
query59	3412	3259	3101	3101
query60	294	270	270	270
query61	91	91	90	90
query62	598	451	443	443
query63	323	295	290	290
query64	8878	2353	1781	1781
query65	3188	3075	3108	3075
query66	755	321	323	321
query67	15327	14913	14923	14913
query68	4487	548	550	548
query69	521	380	334	334
query70	1169	1150	1109	1109
query71	411	285	278	278
query72	7507	5763	5592	5592
query73	749	332	327	327
query74	6121	5583	5468	5468
query75	3320	2624	2651	2624
query76	2349	909	899	899
query77	439	289	300	289
query78	10339	9865	9738	9738
query79	2668	523	518	518
query80	2325	476	466	466
query81	583	223	219	219
query82	756	107	101	101
query83	308	172	168	168
query84	268	89	84	84
query85	1990	295	304	295
query86	488	330	313	313
query87	3277	3076	3054	3054
query88	4110	2426	2431	2426
query89	464	368	379	368
query90	1679	193	192	192
query91	129	99	99	99
query92	62	50	57	50
query93	1826	501	504	501
query94	1037	185	187	185
query95	403	318	310	310
query96	590	271	271	271
query97	3162	3033	3020	3020
query98	217	200	200	200
query99	1199	847	825	825
Total cold run time: 271223 ms
Total hot run time: 172813 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.3 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a37e32bdda3bcadcc6248db46c1b3088e919294b, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.05	0.05
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.50	0.48	0.47
query6	1.14	0.73	0.74
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.50	0.49
query10	0.54	0.54	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.60	0.59	0.60
query14	0.75	0.78	0.78
query15	0.84	0.83	0.82
query16	0.37	0.36	0.35
query17	0.95	1.01	1.03
query18	0.23	0.26	0.22
query19	1.90	1.69	1.69
query20	0.01	0.00	0.01
query21	15.43	0.66	0.66
query22	4.71	7.30	1.66
query23	18.21	1.33	1.28
query24	2.11	0.22	0.22
query25	0.16	0.07	0.08
query26	0.29	0.17	0.17
query27	0.07	0.08	0.07
query28	13.24	1.01	1.01
query29	12.87	3.29	3.27
query30	0.26	0.06	0.06
query31	2.86	0.39	0.38
query32	3.28	0.47	0.47
query33	2.88	2.93	2.90
query34	17.18	4.45	4.41
query35	4.47	4.50	4.45
query36	0.64	0.45	0.49
query37	0.18	0.15	0.15
query38	0.15	0.15	0.15
query39	0.05	0.03	0.04
query40	0.19	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.19 s
Total hot run time: 30.3 s

@englefly
Copy link
Contributor Author

run p0

@morrySnow morrySnow merged commit a481d79 into apache:master Jun 12, 2024
27 of 30 checks passed
dataroaring pushed a commit that referenced this pull request Jun 13, 2024
@englefly @dataroaring
[opt](nereids) using mv's derived stats (#35721)
153fcb6 
for a given mv, we have 2 stats. one is generated by stats analyzer,
which is accurate, another is derived stats, which is derived when
optimizer runs the mv query.
If a query tree contains a mv, we will use the mv's derived stats to
derive stats for the whole tree.
seawinde pushed a commit to seawinde/doris that referenced this pull request Jun 14, 2024
@englefly @seawinde
[opt](nereids) using mv's derived stats (apache#35721)
3fcbbaf 
for a given mv, we have 2 stats. one is generated by stats analyzer,
which is accurate, another is derived stats, which is derived when
optimizer runs the mv query.
If a query tree contains a mv, we will use the mv's derived stats to
derive stats for the whole tree.
englefly added a commit to englefly/incubator-doris that referenced this pull request Jul 1, 2024
@englefly
[opt](nereids) using mv's derived stats (apache#35721)
88f95d0 
for a given mv, we have 2 stats. one is generated by stats analyzer,
which is accurate, another is derived stats, which is derived when
optimizer runs the mv query.
If a query tree contains a mv, we will use the mv's derived stats to
derive stats for the whole tree.

(cherry picked from commit a481d79)
englefly added a commit that referenced this pull request Jul 2, 2024
@englefly
[opt](nereids)using mv's derived stats #35721 (#37098)
0c89346 
pick from #35721

(cherry picked from commit a481d79)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@seawinde seawinde seawinde approved these changes

@starocean999 starocean999 starocean999 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@englefly @doris-robot @morrySnow @starocean999 @seawinde @wm1581066 @dataroaring