Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Split the aggregated results and grouped results into chunks. #20632

Closed
wants to merge 57 commits into from
Closed

Split the aggregated results and grouped results into chunks. #20632

wants to merge 57 commits into from

Conversation

m-schen
Copy link
Contributor

@m-schen m-schen commented Dec 6, 2024 •
edited
Loading

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

https://github.com/matrixorigin/MO-Cloud/issues/4084
https://github.com/matrixorigin/MO-Cloud/issues/3831
#17841
Maybe Fix: #14675

What this PR does / why we need it:

  1. 根据不同类型使用固定的行数对Agg的结果进行拆分，避免发生单个内存超过1G的错误。（字符串类型目前是32768行拆分一次，仍旧存在超过1g的可能性，这个数字是根据目前单行varchar数据不超过16kb来的）。
  2. 简化Group, MergeGroup算子的逻辑。group列的拆分逻辑保持与Agg一致。
  3. 删除AggExec包下多余的代码。

m-schen added 30 commits November 11, 2024 19:05
@m-schen
对agg结果进行拆分, step 0
dfcf519
@m-schen
fix
6b83e1f
@m-schen
fix bug and add ut
9ed27ec
@m-schen
Merge remote-tracking branch 'origin/2.0-dev' into 2.0-1111
690bc73
@m-schen
fmt and more codes
6655f70
@m-schen
a
5abe621
@m-schen
add more
3fb5b39
@m-schen
first version can start
dcc8dce
@m-schen
fix bugs
ff88960
@m-schen
add marshal and unmarshal
3c3cb6a
@m-schen
Merge remote-tracking branch 'origin/2.0-dev' into agg-split
2ba4bc4
@m-schen
use new result to instead of old agg result
9df15f4
@m-schen
remove more
dbbc018
@m-schen
fix count
d9f84fd
@m-schen
add ut and fix bugs
553b66f
@m-schen
add ut and fix dead loop
3b777b1
@m-schen
add more ut
3f5c893
@m-schen
Merge remote-tracking branch 'origin/2.0-dev' into agg-split
0dc3876
@m-schen
fix a flush bug if agg take a group context
58714ae
@m-schen
some rename
491bcac
@m-schen
code clean
685823d
@m-schen
interface rename
51c030d
@m-schen
temp for myself
0b05bfb
@m-schen
fix a wrong return for time window function
b57b81a
@m-schen
add more interface to aggexec for group and mergeGroup operator
4b97cf0
@m-schen
remove unused mergeGroup codes
10adbef
@m-schen
support a flexible merge-group
5037f42
@m-schen
Merge remote-tracking branch 'origin/2.0-dev' into agg-split
bc51632
@m-schen
more methods
b0ef819
@m-schen
more
e686bca
@matrix-meow matrix-meow added the size/XXL Denotes a PR that changes 2000+ lines label Dec 6, 2024
@m-schen
Copy link
Contributor Author

m-schen commented Dec 10, 2024

将pick到main上，暂关闭该pull request.

@m-schen m-schen closed this Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@badboynt1 badboynt1 badboynt1 approved these changes

@ouyuanning ouyuanning ouyuanning approved these changes

@zhangxu19830126 zhangxu19830126 zhangxu19830126 approved these changes

@aunjgr aunjgr Awaiting requested review from aunjgr aunjgr is a code owner

@qingxinhome qingxinhome Awaiting requested review from qingxinhome qingxinhome is a code owner

@sukki37 sukki37 Awaiting requested review from sukki37

Assignees
No one assigned
Labels
do-not-merge/wip kind/bug Something isn't working kind/enhancement size/XXL Denotes a PR that changes 2000+ lines
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@m-schen @zhangxu19830126 @ouyuanning @badboynt1 @matrix-meow