planner, executor: enable inline projection for Limit #20288

pingyu · 2020-09-28T17:10:44Z

What problem does this PR solve?

Issue Number: part of #14428

Problem Summary:
Implement inline projection for Limit, part of #14428.

What is changed and how it works?

What's Changed:
Enable inline projection for LIMIT.

How it Works:
markChildrenUsedCols is applied to Limit.

Benchmark:

$ go test  -v -benchmem -run=XX -bench="BenchmarkLimitExec" -count 1
goos: darwin
goarch: amd64
pkg: github.com/pingcap/tidb/executor
BenchmarkLimitExec/(rows:30000,_offset:10000,_count:10000,_inline_projection:false)-12         	   20329	     58699 ns/op	   24373 B/op	     100 allocs/op
BenchmarkLimitExec/(rows:30000,_offset:10000,_count:10000,_inline_projection:true)-12          	  255342	      5157 ns/op	    3216 B/op	      34 allocs/op
PASS
ok  	github.com/pingcap/tidb/executor	60.073s

About 10x faster for a simple select i from t where j=1 limit 10000,10000
(i & j are both LongLong fields)

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
No.
Need to cherry-pick to the release branch
No.

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

mysql root@localhost:test1> explain select i from t1 where k=1;                                                                                                                                               
+-----------------------+----------+-----------+---------------+--------------------------------+
| id                    | estRows  | task      | access object | operator info                  |
+-----------------------+----------+-----------+---------------+--------------------------------+
| Projection_4          | 10.00    | root      |               | test1.t1.i                     |
| └─TableReader_7       | 10.00    | root      |               | data:Selection_6               |
|   └─Selection_6       | 10.00    | cop[tikv] |               | eq(test1.t1.k, 1)              |
|     └─TableFullScan_5 | 10000.00 | cop[tikv] | table:t1      | keep order:false, stats:pseudo |
+-----------------------+----------+-----------+---------------+--------------------------------+
4 rows in set
Time: 0.014s
mysql root@localhost:test1> explain select i from t1 where k=1 limit 3;                                                                                                                                       
+--------------------------+---------+-----------+---------------+--------------------------------+
| id                       | estRows | task      | access object | operator info                  |
+--------------------------+---------+-----------+---------------+--------------------------------+
| Limit_8                  | 3.00    | root      |               | offset:0, count:3              |
| └─TableReader_14         | 3.00    | root      |               | data:Limit_13                  |
|   └─Limit_13             | 3.00    | cop[tikv] |               | offset:0, count:3              |
|     └─Selection_12       | 3.00    | cop[tikv] |               | eq(test1.t1.k, 1)              |
|       └─TableFullScan_11 | 3000.00 | cop[tikv] | table:t1      | keep order:false, stats:pseudo |
+--------------------------+---------+-----------+---------------+--------------------------------+
5 rows in set
Time: 0.014s
mysql root@localhost:test1>

Side effects

Performance regression
- Consumes more CPU
  No.
- Consumes more MEM
  No.
Breaking backward compatibility
No.

Release note

Support inline projection for Limit

…r_limit_inline_proj

pingyu · 2020-09-28T17:15:52Z

/run-all-tests

pingyu · 2020-09-28T18:59:06Z

/run-all-tests

planner/core/logical_plans.go

planner/core/physical_plans.go

planner/core/rule_column_pruning.go

planner/core/logical_plans.go

util/chunk/chunk.go

pingyu · 2020-10-11T14:55:49Z

/run-all-tests

pingyu · 2020-10-11T15:05:17Z

/run-unit-test

pingyu · 2020-10-11T15:44:34Z

All comments are addressed or replied, PTAL~ @SunRunAway

(Failure of unit-test was caused by infosync.GetAllServerInfo timeout in domain_test.go, and should not be relevant to this PR)

[2020-10-11T15:11:41.828Z] === RUN   TestInfo
...
[2020-10-11T15:11:41.829Z] [2020/10/11 23:10:16.965 +08:00] [WARN] [syncer.go:305] ["[ddl] etcd-cli delete key failed"] [key=/tidb/server/info/957e7044-c832-467c-827d-39ebacf3dc20] [error="context deadline exceeded"] [retryCnt=0]
[2020-10-11T15:11:41.829Z] [2020/10/11 23:10:18.409 +08:00] [INFO] [info.go:773] ["get key failed"] [key=/tidb/server/info] [error="context deadline exceeded"]
[2020-10-11T15:11:41.829Z] [2020/10/11 23:10:19.609 +08:00] [INFO] [info.go:773] ["get key failed"] [key=/tidb/server/info] [error="context deadline exceeded"]
[2020-10-11T15:11:41.829Z] [2020/10/11 23:10:20.810 +08:00] [INFO] [info.go:773] ["get key failed"] [key=/tidb/server/info] [error="context deadline exceeded"]
[2020-10-11T15:11:41.829Z] [2020/10/11 23:10:22.010 +08:00] [INFO] [info.go:773] ["get key failed"] [key=/tidb/server/info] [error="context deadline exceeded"]
[2020-10-11T15:11:41.829Z] [2020/10/11 23:10:23.211 +08:00] [INFO] [info.go:773] ["get key failed"] [key=/tidb/server/info] [error="context deadline exceeded"]
...
[2020-10-11T15:11:41.829Z] --- FAIL: TestInfo (13.08s)
[2020-10-11T15:11:41.829Z]     domain_test.go:220: err context deadline exceeded, infos map[]
[2020-10-11T15:11:41.829Z] FAIL
[2020-10-11T15:11:41.829Z] coverage: 83.3% of statements
[2020-10-11T15:11:41.829Z] FAIL	github.com/pingcap/tidb/domain	42.953s

planner/core/rule_build_key_info.go

SunRunAway · 2020-10-15T05:53:27Z

/reward 750

ti-challenge-bot · 2020-10-15T05:53:30Z

This PR's linked issue is not picked.

SunRunAway · 2020-10-15T06:37:37Z

/cc @XuHuaiyu

SunRunAway · 2020-10-15T07:23:51Z

/reward 750

ti-challenge-bot · 2020-10-15T07:23:56Z

Reward success.

SunRunAway · 2020-10-15T11:55:03Z

/cc @qw4990

qw4990

LGTM

qw4990 · 2020-10-15T12:50:48Z

/merge

ti-srebot · 2020-10-15T12:53:45Z

/run-all-tests

ti-srebot · 2020-10-15T13:05:12Z

@pingyu merge failed.

pingyu · 2020-10-18T03:11:20Z

/run-all-tests

XuHuaiyu · 2020-10-19T02:48:03Z

/merge

ti-srebot · 2020-10-19T02:48:12Z

/run-all-tests

ti-challenge-bot · 2020-10-19T02:55:17Z

@pingyu, Congratulations, you get 750 in this PR, and your total score is 750 in high-performance challenge program.

More

Tip : None

Warning:
The pull request merged, pingyu got the score. But it seems linked issue not picked.

cc: Mentor @SunRunAway

…p#20288)" This reverts commit 4501f6d.

pingyu added 9 commits September 23, 2020 02:26

executor: inline projection for limit

bed804b

wip pingcap#14428

f92d7fa

Merge branch 'master' of https://github.com/pingcap/tidb into executo…

e10c5bc

…r_limit_inline_proj

improve codes

a60340c

Merge branch 'master' into executor_limit_inline_proj

fcf637d

improve codes

f5d1283

fix unittest

165d463

Merge branch 'master' of https://github.com/pingcap/tidb into executo…

f9534d7

…r_limit_inline_proj

improve codes; add benchmark

c17d53d

pingyu requested review from a team as code owners September 28, 2020 17:10

pingyu requested review from SunRunAway and removed request for a team September 28, 2020 17:10

ti-srebot added the contribution This PR is from a community contributor. label Sep 28, 2020

improve benchmark

8bc17b1

github-actions bot added the sig/execution SIG execution label Sep 28, 2020

bug fix. add unit tests

21f49f5

Rustin170506 mentioned this pull request Sep 29, 2020

challenge bot should not report error when the challenge program is not reward ti-community-infra/ti-challenge-bot#150

Closed

1 task

SunRunAway reviewed Oct 10, 2020

View reviewed changes

pingyu added 3 commits October 10, 2020 20:58

Merge branch 'master' into executor_limit_inline_proj

03434d4

address comments

919354c

improve BuildKeyInfo implementation

de436ef

This comment has been minimized.

Sign in to view

SunRunAway reviewed Oct 12, 2020

View reviewed changes

planner/core/rule_build_key_info.go Show resolved Hide resolved

try chunk.Prune

fd47280

ti-srebot requested a review from XuHuaiyu October 15, 2020 06:37

ti-challenge-bot bot added the rewarded label Oct 15, 2020

SunRunAway mentioned this pull request Oct 15, 2020

Support inline projection for executors #14428

Open

9 tasks

ti-srebot requested a review from qw4990 October 15, 2020 11:55

qw4990 approved these changes Oct 15, 2020

View reviewed changes

ti-srebot added status/can-merge Indicates a PR has been approved by a committer. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Oct 15, 2020

ti-srebot approved these changes Oct 15, 2020

View reviewed changes

ti-srebot added the status/LGT2 Indicates that a PR has LGTM 2. label Oct 15, 2020

Merge branch 'master' into executor_limit_inline_proj

4e91cfd

Merge branch 'master' into executor_limit_inline_proj

e756808

ti-srebot merged commit 4501f6d into pingcap:master Oct 19, 2020

pingyu mentioned this pull request Oct 20, 2020

planner, executor: enable inline projection for Selection #20335

Closed

francis0407 mentioned this pull request Nov 3, 2020

planner: support stable result mode #20765

Closed

bb7133 added a commit to bb7133/tidb that referenced this pull request May 14, 2021

Revert "planner, executor: enable inline projection for Limit (pingca…

4b53b35

…p#20288)" This reverts commit 4501f6d.

mjonss mentioned this pull request May 17, 2021

Wrong value when query a partition table with LIMIT #24636

Closed

eurekaka mentioned this pull request Jun 10, 2021

panic may happen when limit is on top of window functions #25344

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner, executor: enable inline projection for Limit #20288

planner, executor: enable inline projection for Limit #20288

pingyu commented Sep 28, 2020 •

edited by SunRunAway

Loading

pingyu commented Sep 28, 2020

pingyu commented Sep 28, 2020

pingyu commented Oct 11, 2020

pingyu commented Oct 11, 2020

pingyu commented Oct 11, 2020

This comment has been minimized.

SunRunAway commented Oct 15, 2020

ti-challenge-bot bot commented Oct 15, 2020

SunRunAway commented Oct 15, 2020

SunRunAway commented Oct 15, 2020

ti-challenge-bot bot commented Oct 15, 2020

SunRunAway commented Oct 15, 2020

qw4990 left a comment

qw4990 commented Oct 15, 2020

ti-srebot commented Oct 15, 2020

ti-srebot commented Oct 15, 2020

pingyu commented Oct 18, 2020

XuHuaiyu commented Oct 19, 2020

ti-srebot commented Oct 19, 2020

ti-challenge-bot bot commented Oct 19, 2020

planner, executor: enable inline projection for Limit #20288

planner, executor: enable inline projection for Limit #20288

Conversation

pingyu commented Sep 28, 2020 • edited by SunRunAway Loading

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

pingyu commented Sep 28, 2020

pingyu commented Sep 28, 2020

pingyu commented Oct 11, 2020

pingyu commented Oct 11, 2020

pingyu commented Oct 11, 2020

This comment has been minimized.

SunRunAway commented Oct 15, 2020

ti-challenge-bot bot commented Oct 15, 2020

SunRunAway commented Oct 15, 2020

SunRunAway commented Oct 15, 2020

ti-challenge-bot bot commented Oct 15, 2020

SunRunAway commented Oct 15, 2020

qw4990 left a comment

Choose a reason for hiding this comment

qw4990 commented Oct 15, 2020

ti-srebot commented Oct 15, 2020

ti-srebot commented Oct 15, 2020

pingyu commented Oct 18, 2020

XuHuaiyu commented Oct 19, 2020

ti-srebot commented Oct 19, 2020

ti-challenge-bot bot commented Oct 19, 2020

pingyu commented Sep 28, 2020 •

edited by SunRunAway

Loading