Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

WSCG failed with join and hash agg #1118

Open
jackylee-ch opened this issue Sep 23, 2022 · 3 comments
Open

WSCG failed with join and hash agg #1118

jackylee-ch opened this issue Sep 23, 2022 · 3 comments
Labels
bug Something isn't working

Comments

@jackylee-ch
Copy link
Contributor

Describe the bug
It seems that it will cause compile failed when there is an aggregation after SMJ.

ArrowColumnarToRow
+- ColumnarHashAggregate(keys=[], functions=[sum(1)])
   +- ArrowCoalesceBatches
      +- ColumnarExchange SinglePartition, ENSURE_REQUIREMENTS, [id=#98], [id=#98], [OUTPUT] List(sum:LongType)
         +- *(1) ColumnarHashAggregate(keys=[], functions=[partial_sum(1)])
            +- *(1) ColumnarSortMergeJoin [tid#7, pid#46], [tid#238, pid#243], Inner, false
@jackylee-ch jackylee-ch added the bug Something isn't working label Sep 23, 2022
@jackylee-ch
Copy link
Contributor Author

It also failed when then is an aggragation, which does not support WSCG, before SMJ. Here is an example.

ColumnarSortMergeJoin [q#5], [q#671], Inner, false
:- ColumnarSort [q#5 ASC NULLS FIRST], false, 0
:  +- xxxxxxx
+- ColumnarConditionProject (isnotnull(s#662) AND isnotnull(q#671))
   +- ColumnarHashAggregate(keys=[q#671], functions=[first_value(s#669, false), sum(p#695L)], output=[q#671, s#662, mp#663L])

@zhouyuan
Copy link
Collaborator

@jackylee-ch

Thanks for debugging, the issue is due to some legacy codegen check in SortMergeJoin - will do clean ups

@jackylee-ch
Copy link
Contributor Author

Thanks for debugging, the issue is due to some legacy codegen check in SortMergeJoin - will do clean ups

@zhouyuan Thanks for reply. I also find compile failed cases when there is an aggregation after SHJ.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants