[SPARK-12085] [SQL] The join condition hidden in DNF can't be pushed down to join operator #10087
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can one of the admins verify this patch? |
|
|
||
| /** | ||
| * Extracts the equal-join condition if any, so that query planner avoids generating cartsian | ||
| * product which cause out of memory exception, and performance issues |
There was a problem hiding this comment.
Can we make this rule simpler and more general? It seems the rule is aptly named, "CNF factorization", and should just do a transformation on expressions (instead of transforming the plan tree and specific to joins). The rest of the rules should then properly fix the join i think.
|
Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch analyzes the filter expression represented as DNF and extract the common equality conditions that match the join relations and expose it out so that the join condition can be pushed down by PushPredicateThroughJoin rule.
With this patch we saw significant performance improvement on TPC-H q19.