[SPARK-18589] [SQL] Fix Python UDF accessing attributes from both side of join #16581
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @hvanhovell |
|
Test build #71346 has finished for PR 16581 at commit
|
|
Test build #71349 has finished for PR 16581 at commit
|
rxin
reviewed
Jan 14, 2017
| case e: SubqueryExpression => | ||
| // non-correlated subquery will be replaced as literal | ||
| e.children.nonEmpty | ||
| case e: Unevaluable => true |
There was a problem hiding this comment.
we need more documentation here on why should be considered evaluable as a join condition.
for example, just looking at this code i have no idea why Uneavaluable is evaluable.
There was a problem hiding this comment.
Unevaluable is not evaluable. This block tries to find a case that is not evaluable in a join, and then negates it by isEmpty. I have to admit that we should document this.
rxin
reviewed
Jan 14, 2017
| @@ -342,6 +342,14 @@ def test_udf_in_filter_on_top_of_outer_join(self): | |||
| df = df.withColumn('b', udf(lambda x: 'x')(df.a)) | |||
| self.assertEqual(df.filter('b = "x"').collect(), [Row(a=1, b='x')]) | |||
|
|
|||
| def test_udf_in_filter_on_top_of_join(self): | |||
davies
commented
Jan 17, 2017
| @@ -342,6 +342,15 @@ def test_udf_in_filter_on_top_of_outer_join(self): | |||
| df = df.withColumn('b', udf(lambda x: 'x')(df.a)) | |||
| self.assertEqual(df.filter('b = "x"').collect(), [Row(a=1, b='x')]) | |||
|
|
|||
| def test_udf_in_filter_on_top_of_join(self): | |||
|
Test build #71520 has finished for PR 16581 at commit
|
|
Test build #3541 has started for PR 16581 at commit |
|
Test build #3542 has finished for PR 16581 at commit
|
|
Test build #71671 has finished for PR 16581 at commit
|
|
Test build #71681 has finished for PR 16581 at commit
|
|
LGTM - merging to master. |
|
I cannot backport it, could open a PR for 2.1. |
asfgit
pushed a commit
that referenced
this pull request
Jan 21, 2017
… of join PythonUDF is unevaluable, which can not be used inside a join condition, currently the optimizer will push a PythonUDF which accessing both side of join into the join condition, then the query will fail to plan. This PR fix this issue by checking the expression is evaluable or not before pushing it into Join. Add a regression test. Author: Davies Liu <[email protected]> Closes #16581 from davies/pyudf_join.
|
Cherry-picked into 2.1 branch. |
uzadude
pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
… of join ## What changes were proposed in this pull request? PythonUDF is unevaluable, which can not be used inside a join condition, currently the optimizer will push a PythonUDF which accessing both side of join into the join condition, then the query will fail to plan. This PR fix this issue by checking the expression is evaluable or not before pushing it into Join. ## How was this patch tested? Add a regression test. Author: Davies Liu <[email protected]> Closes apache#16581 from davies/pyudf_join.
cmonkey
pushed a commit
to cmonkey/spark
that referenced
this pull request
Feb 15, 2017
… of join ## What changes were proposed in this pull request? PythonUDF is unevaluable, which can not be used inside a join condition, currently the optimizer will push a PythonUDF which accessing both side of join into the join condition, then the query will fail to plan. This PR fix this issue by checking the expression is evaluable or not before pushing it into Join. ## How was this patch tested? Add a regression test. Author: Davies Liu <[email protected]> Closes apache#16581 from davies/pyudf_join.
asfgit
pushed a commit
that referenced
this pull request
Sep 27, 2018
… of join in join conditions ## What changes were proposed in this pull request? Thanks for bahchis reporting this. It is more like a follow up work for #16581, this PR fix the scenario of Python UDF accessing attributes from both side of join in join condition. ## How was this patch tested? Add regression tests in PySpark and `BatchEvalPythonExecSuite`. Closes #22326 from xuanyuanking/SPARK-25314. Authored-by: Yuanjian Li <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 2a8cbfd) Signed-off-by: Wenchen Fan <[email protected]>
asfgit
pushed a commit
that referenced
this pull request
Sep 27, 2018
… of join in join conditions ## What changes were proposed in this pull request? Thanks for bahchis reporting this. It is more like a follow up work for #16581, this PR fix the scenario of Python UDF accessing attributes from both side of join in join condition. ## How was this patch tested? Add regression tests in PySpark and `BatchEvalPythonExecSuite`. Closes #22326 from xuanyuanking/SPARK-25314. Authored-by: Yuanjian Li <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
daspalrahul
pushed a commit
to daspalrahul/spark
that referenced
this pull request
Sep 29, 2018
… of join in join conditions ## What changes were proposed in this pull request? Thanks for bahchis reporting this. It is more like a follow up work for apache#16581, this PR fix the scenario of Python UDF accessing attributes from both side of join in join condition. ## How was this patch tested? Add regression tests in PySpark and `BatchEvalPythonExecSuite`. Closes apache#22326 from xuanyuanking/SPARK-25314. Authored-by: Yuanjian Li <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
PythonUDF is unevaluable, which can not be used inside a join condition, currently the optimizer will push a PythonUDF which accessing both side of join into the join condition, then the query will fail to plan.
This PR fix this issue by checking the expression is evaluable or not before pushing it into Join.
How was this patch tested?
Add a regression test.