Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-13698][SQL] Fix Analysis Exceptions when Using Backticks in Generate #11538

Closed
wants to merge 4 commits into from
Closed

[SPARK-13698][SQL] Fix Analysis Exceptions when Using Backticks in Generate #11538

wants to merge 4 commits into from

Conversation

dilipbiswal
Copy link
Contributor

What changes were proposed in this pull request?

Analysis exception occurs while running the following query.

SELECT ints FROM nestedArray LATERAL VIEW explode(a.b) `a` AS `ints`

Failed to analyze query: org.apache.spark.sql.AnalysisException: cannot resolve '`ints`' given input columns: [a, `ints`]; line 1 pos 7
'Project ['ints]
+- Generate explode(a#0.b), true, false, Some(a), [`ints`#8]
   +- SubqueryAlias nestedarray
      +- LocalRelation [a#0], [[[[1,2,3]]]]

How was this patch tested?

Added new unit tests in SQLQuerySuite and HiveQlSuite

@dilipbiswal
Copy link
Contributor Author

cc @liancheng @gatorsmile

@liancheng
Copy link
Contributor

ok to test

@liancheng
Copy link
Contributor

This change itself LGTM. However, the current implementation of cleanIdentifier doesn't handle escaped back-ticks within identifier names, e.g. weirdname``. (Back-ticks within identifiers are escaped by double-back-ticks.) But we can address this issue in another PR.

@SparkQA
Copy link

SparkQA commented Mar 5, 2016

Test build #52516 has finished for PR 11538 at commit 385f6d4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dilipbiswal
Copy link
Contributor Author

@liancheng Thanks a lot. I will work on the follow up PR to handle escaped back-ticks.

@gatorsmile gatorsmile mentioned this pull request Mar 7, 2016
Closed
@cloud-fan
Copy link
Contributor

hi @dilipbiswal , can you update your test case to use escaped back-ticks(e.g. weirdname``)? I think it's already handled by ANTLR.

@dilipbiswal
Copy link
Contributor Author

Hi @cloud-fan Sure.. will do.

@SparkQA
Copy link

SparkQA commented Mar 8, 2016

Test build #52655 has finished for PR 11538 at commit 07af901.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Mar 8, 2016

Test build #52660 has finished for PR 11538 at commit ba6411a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

cloud-fan
cloud-fan reviewed Mar 8, 2016
View reviewed changes
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveQlSuite.scala
|FROM `default`.`src`
|LATERAL VIEW explode(array(array(1, 2, 3))) `gentab1` AS `gencol1`
|LATERAL VIEW explode(`gentab1`.`gencol1`) `gentab2` AS `gencol2`
""".stripMargin)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: please follow the code style of other tests, e.g. https://github.com/dilipbiswal/spark/blob/SPARK-13698/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveQlSuite.scala#L195-L199

@cloud-fan
Copy link
Contributor

LGTM except one style comment

@SparkQA
Copy link

SparkQA commented Mar 9, 2016

Test build #52712 has finished for PR 11538 at commit 7ed8ef7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

LGTM cc @liancheng

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@asfgit asfgit closed this in 53ba6d6 Mar 9, 2016
roygao94 pushed a commit to roygao94/spark that referenced this pull request Mar 22, 2016
@dilipbiswal @roygao94
[SPARK-13698][SQL] Fix Analysis Exceptions when Using Backticks in Ge…
38d245f 
…nerate

## What changes were proposed in this pull request?
Analysis exception occurs while running the following query.
```
SELECT ints FROM nestedArray LATERAL VIEW explode(a.b) `a` AS `ints`
```
```
Failed to analyze query: org.apache.spark.sql.AnalysisException: cannot resolve '`ints`' given input columns: [a, `ints`]; line 1 pos 7
'Project ['ints]
+- Generate explode(a#0.b), true, false, Some(a), [`ints`apache#8]
   +- SubqueryAlias nestedarray
      +- LocalRelation [a#0], [[[[1,2,3]]]]
```

## How was this patch tested?

Added new unit tests in SQLQuerySuite and HiveQlSuite

Author: Dilip Biswal <[email protected]>

Closes apache#11538 from dilipbiswal/SPARK-13698.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dilipbiswal @liancheng @SparkQA @cloud-fan