planner: support cost model for tiflash table scan #12868

lzmhhh123 · 2019-10-22T07:48:45Z

What problem does this PR solve?

Add a new table scan cost in the cost model to distinguish TiKV and TiFlash. Also, generate the TiFlash access path when a table has the TiFlash replica.

What is changed and how it works?

There are four steps:

When a table has at least one TiFlash replica, generate a new access path for it. And don't do skyline pruning for it.
When calculating the cost of the scan operation for a physical plan, we should add the key size for TiKV to distinguish the different row size of TiKV and TiFlash.
Due to the column format of the TiFlash storing arrange. The seek time of a range scan is more than TiKV. TiKV has only one seek time for a range, but TiFlash has len(Columns) times. Then we add another cost to deal with the difference.
Avoid build table scan of TiFlash as the inner child of the index join.

Check List

Tests

Unit test
Integration test

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change
Has persistent data change

Side effects

Possible performance regression
Increased code complexity

codecov · 2019-10-23T08:17:03Z

Codecov Report

Merging #12868 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #12868   +/-   ##
===========================================
  Coverage   80.4636%   80.4636%           
===========================================
  Files           465        465           
  Lines        109514     109514           
===========================================
  Hits          88119      88119           
  Misses        14889      14889           
  Partials       6506       6506

eurekaka

Can you extract the changes of accounting key prefix size of TiKV and related test result changes into a separate PR for easier review and commit management?

planner/core/find_best_task.go

…hh123/tidb into dev/support_cbo_for_tiflash

planner/core/find_best_task.go

alivxxx

LGTM

eurekaka · 2019-10-28T09:16:29Z

cmd/explaintest/r/access_path_selection.result

-  └─IndexScan_11	10000.00	cop[tikv]	table:access_path_selection, index:a, b, range:[NULL,+inf], keep order:false, stats:pseudo
+TableReader_7	3323.33	root	data:Selection_6
+└─Selection_6	3323.33	cop[tikv]	lt(Column#2, 3)
+  └─TableScan_5	10000.00	cop[tikv]	table:access_path_selection, range:[-inf,+inf], keep order:false, stats:pseudo


Looks like many cases in this file are meaningless now.

It's somehow meaningful compared with other explain tests in the file.

sessionctx/variable/session.go

statistics/table.go

planner/core/find_best_task.go

…hh123/tidb into dev/support_cbo_for_tiflash

statistics/table.go

…hh123/tidb into dev/support_cbo_for_tiflash

eurekaka

LGTM

…_tiflash

sre-bot · 2019-10-29T08:02:50Z

/run-all-tests

support cost model for tiflash table scan

89426ab

lzmhhh123 added sig/planner SIG: Planner type/new-feature labels Oct 22, 2019

Merge branch 'master' into dev/support_cbo_for_tiflash

ee30bf0

lzmhhh123 added status/WIP and removed status/WIP labels Oct 22, 2019

lzmhhh123 added 2 commits October 23, 2019 13:04

fix ci

4d7afc3

merge master and fix ci

0029d66

lzmhhh123 force-pushed the dev/support_cbo_for_tiflash branch from 0c206fc to 0029d66 Compare October 23, 2019 08:09

lzmhhh123 requested a review from a team as a code owner October 23, 2019 08:09

ghost requested review from wshwsh12 and removed request for a team October 23, 2019 08:10

pingcap deleted a comment from CLAassistant Oct 23, 2019

Merge branch 'master' into dev/support_cbo_for_tiflash

b1d998f

lzmhhh123 requested review from eurekaka, winoros and alivxxx October 23, 2019 08:14

lzmhhh123 removed the request for review from wshwsh12 October 23, 2019 10:12

eurekaka reviewed Oct 23, 2019

View reviewed changes

Merge branch 'master' into dev/support_cbo_for_tiflash

cbf8060

alivxxx reviewed Oct 25, 2019

View reviewed changes

planner/core/find_best_task.go Outdated Show resolved Hide resolved

lzmhhh123 added 2 commits October 25, 2019 17:55

address comment

161154b

Merge branch 'dev/support_cbo_for_tiflash' of https://github.com/lzmh…

37f1d06

…hh123/tidb into dev/support_cbo_for_tiflash

lzmhhh123 requested a review from alivxxx October 28, 2019 05:39

Merge branch 'master' into dev/support_cbo_for_tiflash

c1f5d15

alivxxx reviewed Oct 28, 2019

View reviewed changes

planner/core/find_best_task.go Show resolved Hide resolved

lzmhhh123 requested a review from alivxxx October 28, 2019 07:24

alivxxx reviewed Oct 28, 2019

View reviewed changes

eurekaka reviewed Oct 28, 2019

View reviewed changes

eurekaka requested a review from wshwsh12 October 28, 2019 11:20

lzmhhh123 added 2 commits October 28, 2019 20:00

address comments

7bb7c7d

Merge branch 'dev/support_cbo_for_tiflash' of https://github.com/lzmh…

bb3d8de

…hh123/tidb into dev/support_cbo_for_tiflash

lzmhhh123 requested a review from eurekaka October 29, 2019 02:13

Merge branch 'master' into dev/support_cbo_for_tiflash

4927c6a

eurekaka reviewed Oct 29, 2019

View reviewed changes

statistics/table.go Outdated Show resolved Hide resolved

lzmhhh123 added 3 commits October 29, 2019 12:15

address comment

922e439

Merge branch 'dev/support_cbo_for_tiflash' of https://github.com/lzmh…

8718aa1

…hh123/tidb into dev/support_cbo_for_tiflash

fix explain test

c97566e

lzmhhh123 requested a review from eurekaka October 29, 2019 04:47

lzmhhh123 added 2 commits October 29, 2019 12:52

check replica available

d2d1a9d

address

b8a6a7a

lzmhhh123 force-pushed the dev/support_cbo_for_tiflash branch from 0fcc6df to b8a6a7a Compare October 29, 2019 07:37

eurekaka previously approved these changes Oct 29, 2019

View reviewed changes

Merge remote-tracking branch 'origin/master' into dev/support_cbo_for…

a399124

…_tiflash

lzmhhh123 dismissed eurekaka’s stale review via a399124 October 29, 2019 07:53

eurekaka approved these changes Oct 29, 2019

View reviewed changes

lzmhhh123 added status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. labels Oct 29, 2019

sre-bot merged commit 6fd74f3 into pingcap:master Oct 29, 2019

lzmhhh123 deleted the dev/support_cbo_for_tiflash branch October 29, 2019 08:07

XiaTianliang pushed a commit to XiaTianliang/tidb that referenced this pull request Dec 21, 2019

planner: support cost model for tiflash table scan (pingcap#12868)

e011895

lzmhhh123 added a commit to lzmhhh123/tidb that referenced this pull request Jan 19, 2020

planner: support cost model for tiflash table scan (pingcap#12868)

507d057

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner: support cost model for tiflash table scan #12868

planner: support cost model for tiflash table scan #12868

lzmhhh123 commented Oct 22, 2019 •

edited

Loading

codecov bot commented Oct 23, 2019 •

edited

Loading

eurekaka left a comment •

edited

Loading

alivxxx left a comment

eurekaka Oct 28, 2019

lzmhhh123 Oct 29, 2019

eurekaka left a comment

sre-bot commented Oct 29, 2019

planner: support cost model for tiflash table scan #12868

planner: support cost model for tiflash table scan #12868

Conversation

lzmhhh123 commented Oct 22, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

codecov bot commented Oct 23, 2019 • edited Loading

Codecov Report

eurekaka left a comment • edited Loading

Choose a reason for hiding this comment

alivxxx left a comment

Choose a reason for hiding this comment

eurekaka Oct 28, 2019

Choose a reason for hiding this comment

lzmhhh123 Oct 29, 2019

Choose a reason for hiding this comment

eurekaka left a comment

Choose a reason for hiding this comment

sre-bot commented Oct 29, 2019

lzmhhh123 commented Oct 22, 2019 •

edited

Loading

codecov bot commented Oct 23, 2019 •

edited

Loading

eurekaka left a comment •

edited

Loading