Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support optional filter in SortMergeJoin #2628

Closed
yjshen opened this issue May 27, 2022 · 2 comments
Closed

Support optional filter in SortMergeJoin #2628

yjshen opened this issue May 27, 2022 · 2 comments
Labels
enhancement New feature or request

Comments

@yjshen
Copy link
Member

yjshen commented May 27, 2022

Align SortMergeJoin with HashJoin for the optional filter support. Based on the efforts in #2509

@yjshen yjshen added the enhancement New feature or request label May 27, 2022
@korowa
Copy link
Contributor

korowa commented May 31, 2022

I've tried to do some POC with constructing intermediate batch and applying filter to it while freeze_buffered_join_streamed -- it seems to be the only place where filtering required, and noticed (please correct me, if I'm mistaken), that due to freeze_* functions logic, output ordering can be broken in case of outer joins -- while freezing, joined and non-joined records from outer table append as separate batches, and after that no merges / resorts happen -- just batch concatenation.

I suppose output ordering to be quite important for planning (i.e. if we had merge/stream/sorted aggregate operator it could be planned over merge join output instead of default HashAggregate), so I wonder - shouldn't this be fixed prior to MJ filter? I guess this fix could significantly change MJ logic in places where filtering required 🤔 -- if so, I can file another issue and dive deeper into MJExec logic.

@yjshen , @richox what do you think of it?

@Dandandan
Copy link
Contributor

AFAIK closed by #9080

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants