Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix](Short Circuit) fix no project list in OlapScanNode #37121

Merged
merged 2 commits into from
Jul 3, 2024

Conversation

eldenmoon
Copy link
Member

query like

select a, `__DORIS_DELETE_SIGN__` from  `test_tbl` WHERE k = '1111111'

will not contain project list on top of OlapScanNode, so for short circuit queries, we should use output exprs on top of plan fragment.

+----------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                            |
|   OUTPUT EXPRS:                                                                                                            |
|     k1[#0]                                                                                                                 |
|     k2[#1]                                                                                                                 |
|     k3[#2]                                                                                                                 |
|     k4[#3]                                                                                                                 |
|     k5[#4]                                                                                                                 |
|     k6[#5]                                                                                                                 |
|     k7[#6]                                                                                                                 |
|     k8[#7]                                                                                                                 |
|     k9[#8]                                                                                                                 |
|     k10[#9]                                                                                                                |
|     k11[#10]                                                                                                               |
|     k12[#11]                                                                                                               |
|     new_column0[#12]                                                                                                       |
|     new_column1[#13]                                                                                                       |
|     __DORIS_DELETE_SIGN__[#14]                                                                                             |
|   PARTITION: UNPARTITIONED                                                                                                 |
|                                                                                                                            |
|   HAS_COLO_PLAN_NODE: false                                                                                                |
|                                                                                                                            |
|   VRESULT SINK                                                                                                             |
|      MYSQL_PROTOCAL                                                                                                        |
|                                                                                                                            |
|   1:VEXCHANGE                                                                                                              |
|      offset: 0                                                                                                             |
|      distribute expr lists: k1[#0], k2[#1], k3[#2]                                                                         |
|                                                                                                                            |
| PLAN FRAGMENT 1                                                                                                            |
|                                                                                                                            |
|   PARTITION: HASH_PARTITIONED: k1[#0], k2[#1], k3[#2]                                                                      |
|                                                                                                                            |
|   HAS_COLO_PLAN_NODE: false                                                                                                |
|                                                                                                                            |
|   STREAM DATA SINK                                                                                                         |
|     EXCHANGE ID: 01                                                                                                        |
|     UNPARTITIONED                                                                                                          |
|                                                                                                                            |
|   0:VOlapScanNode(149)                                                                                                     |
|      TABLE: regression_test_serving_p0.tbl_point_query0(tbl_point_query0), PREAGGREGATION: ON                              |
|      PREDICATES: (k1[#0] = 1231) AND (k2[#1] = 119291.110000000) AND (k3[#2] = 'ddd') AND (__DORIS_DELETE_SIGN__[#14] = 0) |
|      partitions=1/1 (tbl_point_query0)                                                                                     |
|      tablets=1/1, tabletList=13203                                                                                         |
|      cardinality=0, avgRowSize=0.0, numNodes=1                                                                             |
|      pushAggOp=NONE                                                                                                        |
|      SHORT-CIRCUIT                                                                                                         |
|                                                                                                                            |
|                                                                                                                            |
| Statistics                                                                                                                 |
|  planed with unknown column statistics                                                                                     |
+----------------------------------------------------------------------------------------------------------------------------+

Proposed changes

Issue Number: close #xxx

query like

```
select a, `__DORIS_DELETE_SIGN__` from  `test_tbl` WHERE k = '1111111'
```
will not contain project list on top of OlapScanNode, so for short circuit queries, we should use output exprs on top of plan fragment.

```
+----------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                            |
|   OUTPUT EXPRS:                                                                                                            |
|     k1[#0]                                                                                                                 |
|     k2[apache#1]                                                                                                                 |
|     k3[apache#2]                                                                                                                 |
|     k4[apache#3]                                                                                                                 |
|     k5[apache#4]                                                                                                                 |
|     k6[apache#5]                                                                                                                 |
|     k7[apache#6]                                                                                                                 |
|     k8[apache#7]                                                                                                                 |
|     k9[apache#8]                                                                                                                 |
|     k10[apache#9]                                                                                                                |
|     k11[apache#10]                                                                                                               |
|     k12[apache#11]                                                                                                               |
|     new_column0[apache#12]                                                                                                       |
|     new_column1[apache#13]                                                                                                       |
|     __DORIS_DELETE_SIGN__[apache#14]                                                                                             |
|   PARTITION: UNPARTITIONED                                                                                                 |
|                                                                                                                            |
|   HAS_COLO_PLAN_NODE: false                                                                                                |
|                                                                                                                            |
|   VRESULT SINK                                                                                                             |
|      MYSQL_PROTOCAL                                                                                                        |
|                                                                                                                            |
|   1:VEXCHANGE                                                                                                              |
|      offset: 0                                                                                                             |
|      distribute expr lists: k1[#0], k2[apache#1], k3[apache#2]                                                                         |
|                                                                                                                            |
| PLAN FRAGMENT 1                                                                                                            |
|                                                                                                                            |
|   PARTITION: HASH_PARTITIONED: k1[#0], k2[apache#1], k3[apache#2]                                                                      |
|                                                                                                                            |
|   HAS_COLO_PLAN_NODE: false                                                                                                |
|                                                                                                                            |
|   STREAM DATA SINK                                                                                                         |
|     EXCHANGE ID: 01                                                                                                        |
|     UNPARTITIONED                                                                                                          |
|                                                                                                                            |
|   0:VOlapScanNode(149)                                                                                                     |
|      TABLE: regression_test_serving_p0.tbl_point_query0(tbl_point_query0), PREAGGREGATION: ON                              |
|      PREDICATES: (k1[#0] = 1231) AND (k2[apache#1] = 119291.110000000) AND (k3[apache#2] = 'ddd') AND (__DORIS_DELETE_SIGN__[apache#14] = 0) |
|      partitions=1/1 (tbl_point_query0)                                                                                     |
|      tablets=1/1, tabletList=13203                                                                                         |
|      cardinality=0, avgRowSize=0.0, numNodes=1                                                                             |
|      pushAggOp=NONE                                                                                                        |
|      SHORT-CIRCUIT                                                                                                         |
|                                                                                                                            |
|                                                                                                                            |
| Statistics                                                                                                                 |
|  planed with unknown column statistics                                                                                     |
+----------------------------------------------------------------------------------------------------------------------------+
```
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40172 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f33f0f146148a41bd432a87c827a9102c9565b38, data reload: false

------ Round 1 ----------------------------------
q1	18163	4608	4498	4498
q2	2496	195	194	194
q3	11525	1190	1073	1073
q4	10392	858	776	776
q5	7688	2693	2688	2688
q6	228	139	141	139
q7	950	607	624	607
q8	9264	2107	2087	2087
q9	9076	6458	6473	6458
q10	8797	3675	3711	3675
q11	466	246	236	236
q12	436	241	231	231
q13	18874	2986	2940	2940
q14	276	227	230	227
q15	516	483	480	480
q16	523	389	376	376
q17	980	695	714	695
q18	8156	7353	7390	7353
q19	4488	1508	1515	1508
q20	656	331	330	330
q21	4977	3256	4007	3256
q22	419	346	345	345
Total cold run time: 119346 ms
Total hot run time: 40172 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4408	4288	4202	4202
q2	380	281	266	266
q3	3054	2800	2749	2749
q4	1858	1563	1614	1563
q5	5228	5239	5304	5239
q6	218	131	132	131
q7	2154	1723	1735	1723
q8	3187	3334	3314	3314
q9	8288	8303	8286	8286
q10	3876	3646	3704	3646
q11	585	498	481	481
q12	827	673	604	604
q13	16562	2984	2981	2981
q14	291	264	249	249
q15	515	484	482	482
q16	466	420	416	416
q17	1771	1468	1465	1465
q18	7741	7543	7316	7316
q19	1674	1528	1516	1516
q20	1985	1794	1750	1750
q21	4856	4693	4657	4657
q22	631	549	546	546
Total cold run time: 70555 ms
Total hot run time: 53582 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172870 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f33f0f146148a41bd432a87c827a9102c9565b38, data reload: false

query1	917	386	378	378
query2	6459	2466	2415	2415
query3	6662	211	218	211
query4	19065	17544	17061	17061
query5	4233	492	482	482
query6	260	172	189	172
query7	4599	297	288	288
query8	307	307	303	303
query9	8497	2398	2372	2372
query10	626	295	299	295
query11	10537	9968	10062	9968
query12	139	87	84	84
query13	1644	370	378	370
query14	10203	6991	7836	6991
query15	275	187	189	187
query16	8013	276	281	276
query17	1872	545	560	545
query18	2053	282	283	282
query19	202	165	160	160
query20	91	84	85	84
query21	217	131	127	127
query22	4309	3964	4048	3964
query23	33718	33186	32967	32967
query24	11875	2833	2824	2824
query25	641	387	382	382
query26	1685	159	157	157
query27	2920	314	321	314
query28	7545	2083	2066	2066
query29	1034	627	622	622
query30	287	154	152	152
query31	961	757	761	757
query32	98	54	56	54
query33	799	311	312	311
query34	897	471	476	471
query35	763	658	615	615
query36	1108	940	932	932
query37	156	77	79	77
query38	2875	2745	2771	2745
query39	882	807	800	800
query40	280	129	130	129
query41	55	56	55	55
query42	128	107	107	107
query43	617	572	569	569
query44	1170	722	731	722
query45	206	167	170	167
query46	1082	691	709	691
query47	1852	1756	1766	1756
query48	382	305	297	297
query49	1182	422	423	422
query50	775	392	392	392
query51	6904	7004	6785	6785
query52	104	98	98	98
query53	365	297	297	297
query54	999	458	473	458
query55	81	81	79	79
query56	312	300	293	293
query57	1172	1058	1086	1058
query58	269	253	370	253
query59	3580	3309	3352	3309
query60	338	283	282	282
query61	98	93	96	93
query62	661	450	452	450
query63	323	297	293	293
query64	9899	2245	1740	1740
query65	3162	3099	3089	3089
query66	1373	362	329	329
query67	15507	14881	14979	14881
query68	8636	561	570	561
query69	703	483	324	324
query70	1098	1131	1093	1093
query71	516	273	285	273
query72	9084	5462	5126	5126
query73	1179	329	324	324
query74	5982	5498	5482	5482
query75	5071	2650	2657	2650
query76	5008	900	893	893
query77	772	312	311	311
query78	10484	9858	9827	9827
query79	7674	517	532	517
query80	1027	481	465	465
query81	554	218	216	216
query82	788	110	106	106
query83	340	173	168	168
query84	266	84	86	84
query85	1366	277	266	266
query86	414	311	306	306
query87	3306	3127	3087	3087
query88	4998	2393	2391	2391
query89	495	388	375	375
query90	2048	187	190	187
query91	134	99	103	99
query92	62	49	48	48
query93	5609	506	502	502
query94	1298	195	189	189
query95	407	320	315	315
query96	618	287	268	268
query97	3135	3015	3043	3015
query98	222	207	197	197
query99	1177	859	850	850
Total cold run time: 296771 ms
Total hot run time: 172870 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.55 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f33f0f146148a41bd432a87c827a9102c9565b38, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.03
query3	0.23	0.05	0.05
query4	1.67	0.06	0.07
query5	0.51	0.49	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.49
query10	0.54	0.54	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.59
query14	0.77	0.79	0.78
query15	0.84	0.81	0.82
query16	0.35	0.37	0.37
query17	1.02	1.04	1.02
query18	0.21	0.26	0.24
query19	1.90	1.84	1.77
query20	0.02	0.01	0.01
query21	15.47	0.75	0.65
query22	4.63	6.46	1.98
query23	18.25	1.44	1.23
query24	2.13	0.24	0.21
query25	0.15	0.09	0.08
query26	0.25	0.18	0.18
query27	0.08	0.08	0.07
query28	13.24	1.02	0.99
query29	12.66	3.27	3.27
query30	0.25	0.06	0.06
query31	2.87	0.38	0.39
query32	3.26	0.48	0.48
query33	2.91	2.85	2.88
query34	17.01	4.39	4.41
query35	4.49	4.42	4.47
query36	0.65	0.49	0.47
query37	0.18	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.16	0.13	0.14
query41	0.10	0.04	0.05
query42	0.05	0.05	0.05
query43	0.05	0.05	0.03
Total cold run time: 109.86 s
Total hot run time: 30.55 s

@wm1581066 wm1581066 added the usercase Important user case type label label Jul 2, 2024
OlapScanNode olapScanNode = (OlapScanNode) planner.getFragments().get(1).getPlanRoot();
if (olapScanNode.getProjectList() != null) {
// project on scan node
exprs.addAll(olapScanNode.getProjectList().stream()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should think about

result sink([#3, #2, #1])
+-- project([#1, #2, #3])
    +-- scan

Copy link
Member Author

@eldenmoon eldenmoon Jul 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, i will refactor some of the code before legacy planner removed

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 3, 2024
Copy link
Contributor

github-actions bot commented Jul 3, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Jul 3, 2024

PR approved by anyone and no changes requested.

Copy link
Contributor

@lxr599 lxr599 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit def48d5 into apache:master Jul 3, 2024
29 of 34 checks passed
@eldenmoon eldenmoon deleted the qp-ds branch July 3, 2024 07:27
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Jul 9, 2024
query like

```
select a, `__DORIS_DELETE_SIGN__` from  `test_tbl` WHERE k = '1111111'
```
will not contain project list on top of OlapScanNode, so for short
circuit queries, we should use output exprs on top of plan fragment.

```
+----------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                            |
|   OUTPUT EXPRS:                                                                                                            |
|     k1[#0]                                                                                                                 |
|     k2[apache#1]                                                                                                                 |
|     k3[apache#2]                                                                                                                 |
|     k4[apache#3]                                                                                                                 |
|     k5[apache#4]                                                                                                                 |
|     k6[apache#5]                                                                                                                 |
|     k7[apache#6]                                                                                                                 |
|     k8[apache#7]                                                                                                                 |
|     k9[apache#8]                                                                                                                 |
|     k10[apache#9]                                                                                                                |
|     k11[apache#10]                                                                                                               |
|     k12[apache#11]                                                                                                               |
|     new_column0[apache#12]                                                                                                       |
|     new_column1[apache#13]                                                                                                       |
|     __DORIS_DELETE_SIGN__[apache#14]                                                                                             |
|   PARTITION: UNPARTITIONED                                                                                                 |
|                                                                                                                            |
|   HAS_COLO_PLAN_NODE: false                                                                                                |
|                                                                                                                            |
|   VRESULT SINK                                                                                                             |
|      MYSQL_PROTOCAL                                                                                                        |
|                                                                                                                            |
|   1:VEXCHANGE                                                                                                              |
|      offset: 0                                                                                                             |
|      distribute expr lists: k1[#0], k2[apache#1], k3[apache#2]                                                                         |
|                                                                                                                            |
| PLAN FRAGMENT 1                                                                                                            |
|                                                                                                                            |
|   PARTITION: HASH_PARTITIONED: k1[#0], k2[apache#1], k3[apache#2]                                                                      |
|                                                                                                                            |
|   HAS_COLO_PLAN_NODE: false                                                                                                |
|                                                                                                                            |
|   STREAM DATA SINK                                                                                                         |
|     EXCHANGE ID: 01                                                                                                        |
|     UNPARTITIONED                                                                                                          |
|                                                                                                                            |
|   0:VOlapScanNode(149)                                                                                                     |
|      TABLE: regression_test_serving_p0.tbl_point_query0(tbl_point_query0), PREAGGREGATION: ON                              |
|      PREDICATES: (k1[#0] = 1231) AND (k2[apache#1] = 119291.110000000) AND (k3[apache#2] = 'ddd') AND (__DORIS_DELETE_SIGN__[apache#14] = 0) |
|      partitions=1/1 (tbl_point_query0)                                                                                     |
|      tablets=1/1, tabletList=13203                                                                                         |
|      cardinality=0, avgRowSize=0.0, numNodes=1                                                                             |
|      pushAggOp=NONE                                                                                                        |
|      SHORT-CIRCUIT                                                                                                         |
|                                                                                                                            |
|                                                                                                                            |
| Statistics                                                                                                                 |
|  planed with unknown column statistics                                                                                     |
+----------------------------------------------------------------------------------------------------------------------------+
```
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Jul 16, 2024
query like

```
select a, `__DORIS_DELETE_SIGN__` from  `test_tbl` WHERE k = '1111111'
```
will not contain project list on top of OlapScanNode, so for short
circuit queries, we should use output exprs on top of plan fragment.

```
+----------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                            |
+----------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                            |
|   OUTPUT EXPRS:                                                                                                            |
|     k1[#0]                                                                                                                 |
|     k2[apache#1]                                                                                                                 |
|     k3[apache#2]                                                                                                                 |
|     k4[apache#3]                                                                                                                 |
|     k5[apache#4]                                                                                                                 |
|     k6[apache#5]                                                                                                                 |
|     k7[apache#6]                                                                                                                 |
|     k8[apache#7]                                                                                                                 |
|     k9[apache#8]                                                                                                                 |
|     k10[apache#9]                                                                                                                |
|     k11[apache#10]                                                                                                               |
|     k12[apache#11]                                                                                                               |
|     new_column0[apache#12]                                                                                                       |
|     new_column1[apache#13]                                                                                                       |
|     __DORIS_DELETE_SIGN__[apache#14]                                                                                             |
|   PARTITION: UNPARTITIONED                                                                                                 |
|                                                                                                                            |
|   HAS_COLO_PLAN_NODE: false                                                                                                |
|                                                                                                                            |
|   VRESULT SINK                                                                                                             |
|      MYSQL_PROTOCAL                                                                                                        |
|                                                                                                                            |
|   1:VEXCHANGE                                                                                                              |
|      offset: 0                                                                                                             |
|      distribute expr lists: k1[#0], k2[apache#1], k3[apache#2]                                                                         |
|                                                                                                                            |
| PLAN FRAGMENT 1                                                                                                            |
|                                                                                                                            |
|   PARTITION: HASH_PARTITIONED: k1[#0], k2[apache#1], k3[apache#2]                                                                      |
|                                                                                                                            |
|   HAS_COLO_PLAN_NODE: false                                                                                                |
|                                                                                                                            |
|   STREAM DATA SINK                                                                                                         |
|     EXCHANGE ID: 01                                                                                                        |
|     UNPARTITIONED                                                                                                          |
|                                                                                                                            |
|   0:VOlapScanNode(149)                                                                                                     |
|      TABLE: regression_test_serving_p0.tbl_point_query0(tbl_point_query0), PREAGGREGATION: ON                              |
|      PREDICATES: (k1[#0] = 1231) AND (k2[apache#1] = 119291.110000000) AND (k3[apache#2] = 'ddd') AND (__DORIS_DELETE_SIGN__[apache#14] = 0) |
|      partitions=1/1 (tbl_point_query0)                                                                                     |
|      tablets=1/1, tabletList=13203                                                                                         |
|      cardinality=0, avgRowSize=0.0, numNodes=1                                                                             |
|      pushAggOp=NONE                                                                                                        |
|      SHORT-CIRCUIT                                                                                                         |
|                                                                                                                            |
|                                                                                                                            |
| Statistics                                                                                                                 |
|  planed with unknown column statistics                                                                                     |
+----------------------------------------------------------------------------------------------------------------------------+
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged reviewed usercase Important user case type label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants