-
Notifications
You must be signed in to change notification settings - Fork 28.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-20245][SQL][minor] pass output to LogicalRelation directly #17552
Conversation
cc @gatorsmile |
@@ -75,13 +75,13 @@ class PathOptionSuite extends DataSourceTest with SharedSQLContext { | |||
|USING ${classOf[TestOptionsSource].getCanonicalName} | |||
|OPTIONS (PATH '/tmp/path') | |||
""".stripMargin) | |||
assert(getPathOption("src") == Some("file:/tmp/path")) | |||
assert(getPathOption("src").map(makeQualifiedPath) == Some(makeQualifiedPath("/tmp/path"))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These changes in this test suite are not related to this PR, just to reinforce it.
Test build #75578 has finished for PR 17552 at commit
|
// exprId in `expectedOutputAttributes`. | ||
// The reason is that, some relations(like parquet) will reconcile attribute names to | ||
// workaround case insensitivity issue. | ||
case (attr, expected) => attr.withExprId(expected.exprId) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It sounds like this logics mentioned in the comments is removed by this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good catch! I found this logic is only useful when converting hive tables to data source tables, so I moved the logic there: https://github.com/apache/spark/pull/17552/files#diff-ee66e11b56c21364760a5ed2b783f863R215
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree.
LGTM except only one comment |
LGTM pending Jenkins. |
Test build #75587 has finished for PR 17552 at commit
|
thanks for the review, merging to master! |
What changes were proposed in this pull request?
Currently
LogicalRelation
has aexpectedOutputAttributes
parameter, which makes it hard to reason about what the actual output is. Like other leaf nodes,LogicalRelation
should also takeoutput
as a parameter, to simplify the logicHow was this patch tested?
existing tests