Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support window Function lead/lag #5671

Merged
merged 43 commits into from
Sep 2, 2022

Conversation

SeaRise
Copy link
Contributor

@SeaRise SeaRise commented Aug 22, 2022

What problem does this PR solve?

Issue Number: close #5579

tidb pr: pingcap/tidb#37366

Problem Summary:

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Support lead/lag push down to TiFlash.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Aug 22, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • mengxin9014
  • windtalker

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 22, 2022
@SeaRise
Copy link
Contributor Author

SeaRise commented Aug 22, 2022

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Aug 22, 2022

Coverage for changed files

Filename                                           Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DataStreams/WindowBlockInputStream.cpp                 169                26    84.62%          23                 0   100.00%         441                98    77.78%         124                28    77.42%
DataStreams/WindowBlockInputStream.h                    23                 2    91.30%          15                 2    86.67%          48                12    75.00%          14                 1    92.86%
Debug/astToExecutor.cpp                                580               132    77.24%          55                 3    94.55%        1573               354    77.50%         584               176    69.86%
Flash/Coprocessor/DAGExpressionAnalyzer.cpp            461               154    66.59%          54                 7    87.04%         978               259    73.52%         324               118    63.58%
Flash/Coprocessor/DAGExpressionAnalyzer.h                3                 1    66.67%           3                 1    66.67%           6                 1    83.33%           0                 0         -
Flash/Coprocessor/DAGQueryBlockInterpreter.cpp         270                94    65.19%          33                 5    84.85%         606               138    77.23%         150                43    71.33%
Flash/Coprocessor/DAGUtils.cpp                         347               135    61.10%          45                 7    84.44%         638               257    59.72%         412               186    54.85%
Flash/Planner/plans/PhysicalWindow.cpp                   8                 0   100.00%           5                 0   100.00%          50                 3    94.00%           2                 0   100.00%
Flash/tests/gtest_window_executor.cpp                  101                21    79.21%           9                 0   100.00%         298                 0   100.00%          30                12    60.00%
Interpreters/WindowDescription.cpp                      28                11    60.71%           6                 0   100.00%          76                24    68.42%          34                12    64.71%
Interpreters/WindowDescription.h                         1                 1     0.00%           1                 1     0.00%          11                11     0.00%           0                 0         -
Interpreters/convertFieldToType.cpp                    199                81    59.30%          12                 5    58.33%         228               119    47.81%         132                64    51.52%
TestUtils/ExecutorTestUtils.cpp                         48                 7    85.42%          15                 1    93.33%         135                11    91.85%          40                 5    87.50%
TestUtils/ExecutorTestUtils.h                            6                 2    66.67%           3                 0   100.00%          18                 6    66.67%           6                 3    50.00%
TestUtils/mockExecutor.h                                12                 4    66.67%          12                 4    66.67%          26                 6    76.92%           0                 0         -
WindowFunctions/IWindowFunction.cpp                    108                33    69.44%          26                 0   100.00%         197                26    86.80%          46                19    58.70%
WindowFunctions/IWindowFunction.h                        2                 0   100.00%           2                 0   100.00%           2                 0   100.00%           0                 0         -
WindowFunctions/WindowFunctionFactory.cpp               19                 9    52.63%           5                 2    60.00%          30                16    46.67%          10                 6    40.00%
WindowFunctions/WindowFunctionFactory.h                  2                 0   100.00%           2                 0   100.00%          10                 0   100.00%           0                 0         -
WindowFunctions/tests/gtest_lead_lag.cpp               121                49    59.50%          12                 0   100.00%         288                 1    99.65%          32                28    12.50%
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                 2508               762    69.62%         338                38    88.76%        5659              1342    76.29%        1940               701    63.87%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18417      8349             54.67%    212412  86364        59.34%

full coverage report (for internal network access only)

@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 24, 2022
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 24, 2022
@SeaRise
Copy link
Contributor Author

SeaRise commented Aug 31, 2022

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Aug 31, 2022

Coverage for changed files

Filename                                           Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DataStreams/WindowBlockInputStream.cpp                 205                42    79.51%          23                 0   100.00%         464                84    81.90%         140                22    84.29%
DataStreams/WindowBlockInputStream.h                    23                 1    95.65%          15                 1    93.33%          48                 9    81.25%          14                 0   100.00%
Debug/astToExecutor.cpp                                604               132    78.15%          54                 3    94.44%        1609               353    78.06%         608               177    70.89%
Flash/Coprocessor/DAGExpressionAnalyzer.cpp            431               121    71.93%          57                 7    87.72%         999               255    74.47%         322               115    64.29%
Flash/Coprocessor/DAGExpressionAnalyzer.h                3                 1    66.67%           3                 1    66.67%           6                 1    83.33%           0                 0         -
Flash/Coprocessor/DAGQueryBlockInterpreter.cpp         266                94    64.66%          33                 5    84.85%         606               138    77.23%         142                41    71.13%
Flash/Coprocessor/DAGUtils.cpp                         347               134    61.38%          45                 7    84.44%         638               255    60.03%         412               185    55.10%
Flash/Planner/plans/PhysicalWindow.cpp                   8                 0   100.00%           5                 0   100.00%          50                 3    94.00%           2                 0   100.00%
Flash/tests/gtest_window_executor.cpp                  114                28    75.44%          10                 0   100.00%         352                 0   100.00%          34                16    52.94%
Interpreters/WindowDescription.cpp                      28                11    60.71%           6                 0   100.00%          76                24    68.42%          34                12    64.71%
Interpreters/WindowDescription.h                         1                 1     0.00%           1                 1     0.00%          11                11     0.00%           0                 0         -
Interpreters/convertFieldToType.cpp                    199                81    59.30%          12                 5    58.33%         228               119    47.81%         132                64    51.52%
TestUtils/ExecutorTestUtils.cpp                         46                 7    84.78%          13                 1    92.31%         124                11    91.13%          40                 5    87.50%
TestUtils/ExecutorTestUtils.h                            3                 1    66.67%           3                 1    66.67%          18                13    27.78%           0                 0         -
TestUtils/mockExecutor.h                                14                 4    71.43%          14                 4    71.43%          32                 6    81.25%           0                 0         -
WindowFunctions/IWindowFunction.cpp                    108                33    69.44%          26                 0   100.00%         195                23    88.21%          46                19    58.70%
WindowFunctions/IWindowFunction.h                        2                 0   100.00%           2                 0   100.00%           2                 0   100.00%           0                 0         -
WindowFunctions/WindowFunctionFactory.cpp               19                 9    52.63%           5                 2    60.00%          30                16    46.67%          10                 6    40.00%
WindowFunctions/WindowFunctionFactory.h                  2                 0   100.00%           2                 0   100.00%          10                 0   100.00%           0                 0         -
WindowFunctions/tests/gtest_lead_lag.cpp               130                49    62.31%          12                 0   100.00%         335                 1    99.70%          34                28    17.65%
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                 2553               749    70.66%         341                38    88.86%        5833              1322    77.34%        1970               690    64.97%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18525      8320             55.09%    214541  85924        59.95%

full coverage report (for internal network access only)

@SeaRise
Copy link
Contributor Author

SeaRise commented Aug 31, 2022

/run-integration-test

@SeaRise
Copy link
Contributor Author

SeaRise commented Sep 1, 2022

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Sep 1, 2022

Coverage for changed files

Filename                                           Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DataStreams/WindowBlockInputStream.cpp                 205                42    79.51%          23                 0   100.00%         464                84    81.90%         140                22    84.29%
DataStreams/WindowBlockInputStream.h                    23                 1    95.65%          15                 1    93.33%          48                 9    81.25%          14                 0   100.00%
Debug/astToExecutor.cpp                                604               132    78.15%          54                 3    94.44%        1616               353    78.16%         608               177    70.89%
Flash/Coprocessor/DAGExpressionAnalyzer.cpp            431               121    71.93%          57                 7    87.72%         999               255    74.47%         322               115    64.29%
Flash/Coprocessor/DAGExpressionAnalyzer.h                3                 1    66.67%           3                 1    66.67%           6                 1    83.33%           0                 0         -
Flash/Coprocessor/DAGQueryBlockInterpreter.cpp         266                94    64.66%          33                 5    84.85%         606               138    77.23%         142                41    71.13%
Flash/Coprocessor/DAGUtils.cpp                         347               134    61.38%          45                 7    84.44%         638               255    60.03%         412               185    55.10%
Flash/Planner/plans/PhysicalWindow.cpp                   8                 0   100.00%           5                 0   100.00%          50                 3    94.00%           2                 0   100.00%
Flash/tests/gtest_window_executor.cpp                  114                28    75.44%          10                 0   100.00%         352                 0   100.00%          34                16    52.94%
Interpreters/WindowDescription.cpp                      28                11    60.71%           6                 0   100.00%          76                24    68.42%          34                12    64.71%
Interpreters/WindowDescription.h                         1                 1     0.00%           1                 1     0.00%          11                11     0.00%           0                 0         -
Interpreters/convertFieldToType.cpp                    199                81    59.30%          12                 5    58.33%         228               119    47.81%         132                64    51.52%
TestUtils/ExecutorTestUtils.cpp                         46                 7    84.78%          13                 1    92.31%         124                11    91.13%          40                 5    87.50%
TestUtils/ExecutorTestUtils.h                            3                 1    66.67%           3                 1    66.67%          18                13    27.78%           0                 0         -
TestUtils/mockExecutor.h                                14                 4    71.43%          14                 4    71.43%          32                 6    81.25%           0                 0         -
WindowFunctions/IWindowFunction.cpp                    111                33    70.27%          26                 0   100.00%         195                23    88.21%          48                19    60.42%
WindowFunctions/IWindowFunction.h                        2                 0   100.00%           2                 0   100.00%           2                 0   100.00%           0                 0         -
WindowFunctions/WindowFunctionFactory.cpp               19                 9    52.63%           5                 2    60.00%          30                16    46.67%          10                 6    40.00%
WindowFunctions/WindowFunctionFactory.h                  2                 0   100.00%           2                 0   100.00%          10                 0   100.00%           0                 0         -
WindowFunctions/tests/gtest_lead_lag.cpp               132                49    62.88%          12                 0   100.00%         347                 1    99.71%          34                28    17.65%
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                 2558               749    70.72%         341                38    88.86%        5852              1322    77.41%        1972               690    65.01%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18570      8327             55.16%    214961  85907        60.04%

full coverage report (for internal network access only)

@SeaRise SeaRise requested a review from windtalker September 1, 2022 03:58
Copy link
Contributor

@windtalker windtalker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 1, 2022
@SeaRise
Copy link
Contributor Author

SeaRise commented Sep 2, 2022

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Sep 2, 2022

Coverage for changed files

Filename                                           Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DataStreams/WindowBlockInputStream.cpp                 205                42    79.51%          23                 0   100.00%         464                84    81.90%         140                22    84.29%
DataStreams/WindowBlockInputStream.h                    23                 1    95.65%          15                 1    93.33%          48                 9    81.25%          14                 0   100.00%
Debug/astToExecutor.cpp                                604               132    78.15%          54                 3    94.44%        1616               353    78.16%         608               177    70.89%
Flash/Coprocessor/DAGExpressionAnalyzer.cpp            431               121    71.93%          57                 7    87.72%         999               255    74.47%         322               115    64.29%
Flash/Coprocessor/DAGExpressionAnalyzer.h                3                 1    66.67%           3                 1    66.67%           6                 1    83.33%           0                 0         -
Flash/Coprocessor/DAGQueryBlockInterpreter.cpp         266                94    64.66%          33                 5    84.85%         606               138    77.23%         142                41    71.13%
Flash/Coprocessor/DAGUtils.cpp                         347               134    61.38%          45                 7    84.44%         638               255    60.03%         412               185    55.10%
Flash/Planner/plans/PhysicalWindow.cpp                   8                 0   100.00%           5                 0   100.00%          50                 3    94.00%           2                 0   100.00%
Flash/tests/gtest_window_executor.cpp                  114                28    75.44%          10                 0   100.00%         352                 0   100.00%          34                16    52.94%
Interpreters/WindowDescription.cpp                      28                11    60.71%           6                 0   100.00%          76                24    68.42%          34                12    64.71%
Interpreters/WindowDescription.h                         1                 1     0.00%           1                 1     0.00%          11                11     0.00%           0                 0         -
Interpreters/convertFieldToType.cpp                    199                81    59.30%          12                 5    58.33%         228               119    47.81%         132                64    51.52%
TestUtils/ExecutorTestUtils.cpp                         46                 7    84.78%          13                 1    92.31%         124                11    91.13%          40                 5    87.50%
TestUtils/ExecutorTestUtils.h                            3                 1    66.67%           3                 1    66.67%          18                13    27.78%           0                 0         -
TestUtils/mockExecutor.h                                14                 4    71.43%          14                 4    71.43%          32                 6    81.25%           0                 0         -
WindowFunctions/IWindowFunction.cpp                    111                33    70.27%          26                 0   100.00%         195                23    88.21%          48                19    60.42%
WindowFunctions/IWindowFunction.h                        2                 0   100.00%           2                 0   100.00%           2                 0   100.00%           0                 0         -
WindowFunctions/WindowFunctionFactory.cpp               19                 9    52.63%           5                 2    60.00%          30                16    46.67%          10                 6    40.00%
WindowFunctions/WindowFunctionFactory.h                  2                 0   100.00%           2                 0   100.00%          10                 0   100.00%           0                 0         -
WindowFunctions/tests/gtest_lead_lag.cpp               132                49    62.88%          12                 0   100.00%         347                 1    99.71%          34                28    17.65%
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                 2558               749    70.72%         341                38    88.86%        5852              1322    77.41%        1972               690    65.01%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18625      8328             55.29%    215386  85832        60.15%

full coverage report (for internal network access only)

Copy link
Contributor

@mengxin9014 mengxin9014 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 2, 2022
@SeaRise
Copy link
Contributor Author

SeaRise commented Sep 2, 2022

/merge

@ti-chi-bot
Copy link
Member

@SeaRise: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 6fa337e

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 2, 2022
@ti-chi-bot
Copy link
Member

@SeaRise: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@sre-bot
Copy link
Collaborator

sre-bot commented Sep 2, 2022

Coverage for changed files

Filename                                           Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DataStreams/WindowBlockInputStream.cpp                 205                42    79.51%          23                 0   100.00%         464                84    81.90%         140                22    84.29%
DataStreams/WindowBlockInputStream.h                    23                 1    95.65%          15                 1    93.33%          48                 9    81.25%          14                 0   100.00%
Debug/astToExecutor.cpp                                604               132    78.15%          54                 3    94.44%        1616               353    78.16%         608               177    70.89%
Flash/Coprocessor/DAGExpressionAnalyzer.cpp            431               121    71.93%          57                 7    87.72%         999               255    74.47%         322               115    64.29%
Flash/Coprocessor/DAGExpressionAnalyzer.h                3                 1    66.67%           3                 1    66.67%           6                 1    83.33%           0                 0         -
Flash/Coprocessor/DAGQueryBlockInterpreter.cpp         266                94    64.66%          33                 5    84.85%         606               138    77.23%         142                41    71.13%
Flash/Coprocessor/DAGUtils.cpp                         347               134    61.38%          45                 7    84.44%         638               255    60.03%         412               185    55.10%
Flash/Planner/plans/PhysicalWindow.cpp                   8                 0   100.00%           5                 0   100.00%          50                 3    94.00%           2                 0   100.00%
Flash/tests/gtest_window_executor.cpp                  114                28    75.44%          10                 0   100.00%         352                 0   100.00%          34                16    52.94%
Interpreters/WindowDescription.cpp                      28                11    60.71%           6                 0   100.00%          76                24    68.42%          34                12    64.71%
Interpreters/WindowDescription.h                         1                 1     0.00%           1                 1     0.00%          11                11     0.00%           0                 0         -
Interpreters/convertFieldToType.cpp                    199                81    59.30%          12                 5    58.33%         228               119    47.81%         132                64    51.52%
TestUtils/ExecutorTestUtils.cpp                         46                 7    84.78%          13                 1    92.31%         124                11    91.13%          40                 5    87.50%
TestUtils/ExecutorTestUtils.h                            3                 1    66.67%           3                 1    66.67%          18                13    27.78%           0                 0         -
TestUtils/mockExecutor.h                                14                 4    71.43%          14                 4    71.43%          32                 6    81.25%           0                 0         -
WindowFunctions/IWindowFunction.cpp                    111                33    70.27%          26                 0   100.00%         195                23    88.21%          48                19    60.42%
WindowFunctions/IWindowFunction.h                        2                 0   100.00%           2                 0   100.00%           2                 0   100.00%           0                 0         -
WindowFunctions/WindowFunctionFactory.cpp               19                 9    52.63%           5                 2    60.00%          30                16    46.67%          10                 6    40.00%
WindowFunctions/WindowFunctionFactory.h                  2                 0   100.00%           2                 0   100.00%          10                 0   100.00%           0                 0         -
WindowFunctions/tests/gtest_lead_lag.cpp               132                49    62.88%          12                 0   100.00%         347                 1    99.71%          34                28    17.65%
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                 2558               749    70.72%         341                38    88.86%        5852              1322    77.41%        1972               690    65.01%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18625      8332             55.26%    215386  85857        60.14%

full coverage report (for internal network access only)

@ti-chi-bot ti-chi-bot merged commit e9f3239 into pingcap:master Sep 2, 2022
JaySon-Huang pushed a commit to JaySon-Huang/tiflash that referenced this pull request Sep 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement lead/lag window function push down
5 participants