Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

expression, planner: allow pushdown count distinct when enumerate physical plans #22867

Merged
merged 24 commits into from
Mar 11, 2021
Merged

expression, planner: allow pushdown count distinct when enumerate physical plans #22867

merged 24 commits into from
Mar 11, 2021

Conversation

tisonkun
Copy link
Contributor

@tisonkun tisonkun commented Feb 22, 2021 •
edited
Loading

What problem does this PR solve?

Issue Number: related to pingcap/tiflash#1428

co pingcap/tiflash#1457

Problem Summary:

support push down count distinct aggregation to tiflash in mpp mode.

What is changed and how it works?

see also pingcap/tiflash#1428

What's Changed:

  • update planner module to enable push down count distinct
  • update expression module to pass HasDistinct attr.

How it Works:

  • straightforward pushing down count distinct with necessary info.

Related changes

N/A

Check List

Tests

  • Unit test
  • Integration test

Side effects

N/A

Release note

  • Support DISTINCT AGG pushdown to TiFlash

@tisonkun tisonkun requested a review from a team as a code owner February 22, 2021 08:42
@tisonkun tisonkun requested review from winoros and removed request for a team February 22, 2021 08:42
@ti-srebot ti-srebot added the first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. label Feb 22, 2021
@sre-bot
Copy link
Contributor

sre-bot commented Feb 22, 2021

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

@CLAassistant
Copy link

CLAassistant commented Feb 22, 2021 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@zhouqiang-cl
Copy link
Contributor

/run-check-release-note

@sre-bot
Copy link
Contributor

sre-bot commented Feb 22, 2021

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

@zhouqiang-cl
Copy link
Contributor

/run-check-release-note

1 similar comment
@zhouqiang-cl
Copy link
Contributor

/run-check-release-note

@sre-bot
Copy link
Contributor

sre-bot commented Feb 22, 2021

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

@zhouqiang-cl
Copy link
Contributor

/run-check-release-note

@ti-srebot
Copy link
Contributor

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

@tisonkun tisonkun marked this pull request as draft February 22, 2021 12:34
@tisonkun tisonkun force-pushed the pushdown-distinct-aggregation branch from 9dd6010 to c314ede Compare February 23, 2021 07:04
@ti-srebot
Copy link
Contributor

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

@ti-chi-bot ti-chi-bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Feb 23, 2021
@ti-srebot
Copy link
Contributor

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

@ti-chi-bot ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 23, 2021
@tisonkun
Copy link
Contributor Author

/run-check-release-note

@tisonkun tisonkun force-pushed the pushdown-distinct-aggregation branch 2 times, most recently from e68fe4e to 47047c5 Compare February 23, 2021 08:22
@tisonkun
Copy link
Contributor Author

ref pingcap/tiflash#1428

@tisonkun
Copy link
Contributor Author

@hanfei1991 please take a look. I'm unsure if we should make changes in PhysicalHashAgg.attach2TaskForMpp in order to only allow Mpp1Phase to go, or others are already rejected by current logic.

So far, the planner will choose Mpp1Phase physical plan. Tests will be added later.

@tisonkun tisonkun force-pushed the pushdown-distinct-aggregation branch from 47047c5 to 8464347 Compare March 4, 2021 02:41
@tisonkun tisonkun mentioned this pull request Mar 4, 2021
Merged
@tisonkun tisonkun force-pushed the pushdown-distinct-aggregation branch from 8464347 to 5aab25c Compare March 5, 2021 02:24
@tisonkun tisonkun changed the title WIP: allow pushdown count distinct when enumerate physical plans allow pushdown count distinct when enumerate physical plans Mar 5, 2021
@tisonkun tisonkun marked this pull request as ready for review March 5, 2021 02:35
@tisonkun tisonkun requested a review from a team as a code owner March 5, 2021 02:35
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 10, 2021
@hanfei1991
Copy link
Member

add a test in planner: select count(distinct **), sum(distinct **) from .... . To prove that the pushdown check really works well

@tisonkun
add failing test
b834726 
Signed-off-by: tison <[email protected]>
@tisonkun tisonkun force-pushed the pushdown-distinct-aggregation branch from 970658b to b834726 Compare March 10, 2021 06:50
@tisonkun
Copy link
Contributor Author

add a test in planner: select count(distinct **), sum(distinct **) from .... . To prove that the pushdown check really works well

add failing logic as b834726

@tisonkun
Copy link
Contributor Author

@hanfei1991 @windtalker PTAL

@tisonkun
Copy link
Contributor Author

@fzhedu please take a look.

fzhedu
fzhedu reviewed Mar 11, 2021
View reviewed changes
planner/core/exhaust_physical_plans.go
// If AllowDistinctAggPushDown is set to true, we should not consider RootTask.
if !la.ctx.GetSessionVars().AllowDistinctAggPushDown {
// TODO: remove after the cost estimation of distinct pushdown is implemented.
if !la.ctx.GetSessionVars().AllowDistinctAggPushDown && !canPushDownToMPP {
Copy link
Contributor

@fzhedu fzhedu Mar 11, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this change is wrong.
If AllowDistinctAggPushDown is set to false, we should only consider RootTask.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also need a test to ensure this.

Copy link
Contributor Author

@tisonkun tisonkun Mar 11, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC @hanfei1991 stands that we don't share the same config with AllowDistinctAggPushDown which is introduced by #15500 and focuses on rewrite distinct in agg into group by.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the "AllowDistinctAggPushDown" doesn't decide whether to push mpp query.

Copy link
Contributor

@fzhedu fzhedu Mar 11, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after disscussed with @zanmato1984 and @hanfei1991 , I approve this change, even through it induces some unexpections.

fzhedu
fzhedu approved these changes Mar 11, 2021
View reviewed changes
Copy link
Contributor

@fzhedu fzhedu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • fzhedu
  • hanfei1991

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 11, 2021
@hanfei1991 hanfei1991 self-assigned this Mar 11, 2021
@hanfei1991
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 4b5cd54

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 11, 2021
@ti-chi-bot
Copy link
Member

@tisonkun: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot merged commit 362883c into pingcap:master Mar 11, 2021
@tisonkun tisonkun mentioned this pull request Mar 12, 2021
Closed
@tisonkun tisonkun deleted the pushdown-distinct-aggregation branch April 17, 2021 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hanfei1991 hanfei1991 hanfei1991 left review comments

@windtalker windtalker windtalker left review comments

@fzhedu fzhedu fzhedu approved these changes

@winoros winoros Awaiting requested review from winoros

@qw4990 qw4990 Awaiting requested review from qw4990

Assignees

@hanfei1991 hanfei1991

Labels
component/expression first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
9 participants
@tisonkun @sre-bot @CLAassistant @zhouqiang-cl @ti-srebot @hanfei1991 @ti-chi-bot @windtalker @fzhedu