planner: support push down broadcast cartesian join to TiFlash #25049

windtalker · 2021-06-02T05:40:56Z

What problem does this PR solve?

Issue Number: close pingcap/tiflash#1938

Problem Summary:

Currently, TiFlash does not support cartesian join, so TiDB will not push down cartesian join to TiFlash.

What is changed and how it works?

Proposal: xxx

What's Changed:

How it Works:

After pingcap/tiflash#2041 TiFlash will support cartesian join, so in TiDB, it will push down a broadcast cartesian join to TiFlash.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Integration test
Manual test (add detailed scripts or steps below)

Side effects

Performance regression
- Consumes more CPU
- Consumes more MEM
Breaking backward compatibility

Release note

support push down broadcast cartesian join to TiFlash

* support cross broadcast join * save work * save work * support push down cartesian join to tiflash * enable cartesian push down by default * refine * fix * fix

…ition is not empty while the join type is not left/right (pingcap#24846) * support cross broadcast join * save work * save work * support push down cartesian join to tiflash * enable cartesian push down by default * refine * fix * fix * fix ci

…an_join

sre-bot · 2021-06-02T05:41:31Z

Please follow PR Title Format:

pkg [, pkg2, pkg3]: what's changed

Or if the count of mainly changed packages are more than 3, use

*: what's changed

…an_join

hanfei1991 · 2021-06-03T06:44:51Z

planner/core/exhaust_physical_plans.go

 		return nil
 	}
+
+	if len(p.EqualConditions) == 0 {
+		if p.ctx.GetSessionVars().AllowCartesianBCJ < 1 || !useBCJ {


== 0 is more direct ?

fzhedu · 2021-06-03T08:38:53Z

planner/core/exhaust_physical_plans.go

+	if (len(p.LeftConditions) != 0 && p.JoinType != LeftOuterJoin) || (len(p.RightConditions) != 0 && p.JoinType != RightOuterJoin) {
+		return nil
+	}


why add this?
a inner join cannot have leftConditions or rightConditions?

Because TiFlash assuming leftConditions only exists for left join and rightConditions only exists for right join

fzhedu · 2021-06-03T08:39:07Z

planner/core/exhaust_physical_plans.go

+	if (len(p.LeftConditions) != 0 && p.JoinType != LeftOuterJoin) || (len(p.RightConditions) != 0 && p.JoinType != RightOuterJoin) {
+		return nil
+	}


why add this?
a inner join cannot have leftConditions or rightConditions?

fzhedu · 2021-06-03T08:46:15Z

sessionctx/variable/session.go

+	// AllowCartesianBCJ means allow broadcast CARTESIAN join, 0 means not allow, 1 means allow broadcast CARTESIAN join
+	// but the table size should under the broadcast threshold, 2 means allow broadcast CARTESIAN join even if the table
+	// size exceeds the broadcast threshold
+	AllowCartesianBCJ int


sessioni level or instance level?

session level.

fzhedu · 2021-06-03T08:46:45Z

planner/core/testdata/integration_serial_suite_in.json

+      "explain format = 'brief' select count(*) from fact_t join d1_t on fact_t.d1_k > d1_t.d1_k",
+      "explain format = 'brief' select count(*) from fact_t left join d1_t on fact_t.d1_k > d1_t.d1_k",
+      "explain format = 'brief' select count(*) from fact_t right join d1_t on fact_t.d1_k > d1_t.d1_k",
+      "explain format = 'brief' select count(*) from fact_t where d1_k not in (select d1_k from d1_t)"


add some example for semi join?

fzhedu · 2021-06-03T08:57:25Z

LGTM

fzhedu · 2021-06-03T08:57:34Z

/merge

ti-chi-bot · 2021-06-03T08:57:37Z

@fzhedu: /merge in this pull request requires 2 approval(s).

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

fzhedu · 2021-06-03T09:02:25Z

/LGTM

ti-chi-bot · 2021-06-03T09:02:26Z

@fzhedu: Please use GitHub review feature instead of /lgtm [cancel] when you want to submit review to the pull request.
For how to use GitHub review feature, see also this document provided by GitHub.

For the reason we drop support to the commands, see also this page.
This reply is being used as a temporary reply during the migration of review process and will be removed on July 1st.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

fzhedu

/LGTM

ti-chi-bot · 2021-06-03T09:04:03Z

@fzhedu: Please use GitHub review feature instead of /lgtm [cancel] when you want to submit review to the pull request.
For how to use GitHub review feature, see also this document provided by GitHub.

For the reason we drop support to the commands, see also this page.
This reply is being used as a temporary reply during the migration of review process and will be removed on July 1st.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-06-03T09:04:05Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

fzhedu
hanfei1991

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

windtalker · 2021-06-03T09:04:29Z

/merge

ti-chi-bot · 2021-06-03T09:04:31Z

This pull request has been accepted and is ready to merge.

Commit hash: f892839

windtalker · 2021-06-03T09:24:19Z

/run-check_dev_2

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2021-06-03T10:36:17Z

cherry pick to release-5.0 in PR #25106

… (#25106)

windtalker and others added 4 commits June 1, 2021 17:00

planner: support pushdown cartesian join to TiFlash (pingcap#24837)

6389486

* support cross broadcast join * save work * save work * support push down cartesian join to tiflash * enable cartesian push down by default * refine * fix * fix

Merge branch 'master' of https://github.com/pingcap/tidb into cartesi…

4896c8b

…an_join

support push down broadcast cartesian join to TiFlash

b7b4a73

windtalker requested a review from a team as a code owner June 2, 2021 05:40

windtalker requested review from rebelice and removed request for a team June 2, 2021 05:40

ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 2, 2021

windtalker changed the title ~~support push down broadcast cartesian join to TiFlash~~ planner: support push down broadcast cartesian join to TiFlash Jun 2, 2021

Merge branch 'master' of https://github.com/pingcap/tidb into cartesi…

1cf6b51

…an_join

Merge branch 'master' of https://github.com/pingcap/tidb into cartesi…

1edb01f

…an_join

hanfei1991 reviewed Jun 3, 2021

View reviewed changes

address comments

f892839

hanfei1991 approved these changes Jun 3, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 3, 2021

fzhedu reviewed Jun 3, 2021

View reviewed changes

fzhedu approved these changes Jun 3, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 3, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 3, 2021

Merge branch 'master' into cartesian_join

440907f

Merge branch 'master' into cartesian_join

c3ec801

ti-chi-bot merged commit d3de547 into pingcap:master Jun 3, 2021

fzhedu added the needs-cherry-pick-release-5.0 label Jun 3, 2021

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jun 3, 2021

cherry pick pingcap#25049 to release-5.0

9be4ac9

Signed-off-by: ti-srebot <[email protected]>

ti-srebot mentioned this pull request Jun 3, 2021

planner: support push down broadcast cartesian join to TiFlash (#25049) #25106

Merged

ti-chi-bot pushed a commit that referenced this pull request Jun 17, 2021

planner: support push down broadcast cartesian join to TiFlash (#25049)…

d233624

… (#25106)

windtalker deleted the cartesian_join branch November 16, 2021 01:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner: support push down broadcast cartesian join to TiFlash #25049

planner: support push down broadcast cartesian join to TiFlash #25049

windtalker commented Jun 2, 2021

sre-bot commented Jun 2, 2021

hanfei1991 Jun 3, 2021

windtalker Jun 3, 2021

fzhedu Jun 3, 2021

windtalker Jun 3, 2021

fzhedu Jun 3, 2021

fzhedu Jun 3, 2021

windtalker Jun 3, 2021

fzhedu Jun 3, 2021

fzhedu commented Jun 3, 2021

fzhedu commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

fzhedu commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

fzhedu left a comment

ti-chi-bot commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

windtalker commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

windtalker commented Jun 3, 2021

ti-srebot commented Jun 3, 2021

planner: support push down broadcast cartesian join to TiFlash #25049

planner: support push down broadcast cartesian join to TiFlash #25049

Conversation

windtalker commented Jun 2, 2021

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

sre-bot commented Jun 2, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fzhedu commented Jun 3, 2021

fzhedu commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

fzhedu commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

fzhedu left a comment

Choose a reason for hiding this comment

ti-chi-bot commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

windtalker commented Jun 3, 2021

ti-chi-bot commented Jun 3, 2021

windtalker commented Jun 3, 2021

ti-srebot commented Jun 3, 2021