[SPARK-22972] Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc #20165

xubo245 · 2018-01-05T13:28:23Z

What changes were proposed in this pull request?

Fix the warning: Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc.

How was this patch tested?

test("SPARK-22972: hive orc source")
assert(HiveSerDe.sourceToSerDe("org.apache.spark.sql.hive.orc")
.equals(HiveSerDe.sourceToSerDe("orc")))

…provider org.apache.spark.sql.hive.orc.

gatorsmile · 2018-01-05T13:38:25Z

ok to test

gatorsmile · 2018-01-05T13:43:27Z

sql/core/src/main/scala/org/apache/spark/sql/internal/HiveSerDe.scala

@@ -72,7 +72,7 @@ object HiveSerDe {
  def sourceToSerDe(source: String): Option[HiveSerDe] = {
    val key = source.toLowerCase(Locale.ROOT) match {
      case s if s.startsWith("org.apache.spark.sql.parquet") => "parquet"
-      case s if s.startsWith("org.apache.spark.sql.orc") => "orc"


We also need to keep the original one. We do not want to introduce behavior breaks.

gatorsmile · 2018-01-05T13:44:35Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala

+         |  PATH '${new File(orcTableAsDir.getAbsolutePath).toURI}'
+         |)
+       """.
+        stripMargin)


combine the line 73 and 74

gatorsmile · 2018-01-05T13:44:52Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala

+       """.
+        stripMargin)
+    spark.sql(
+      "desc formatted normal_orc_as_source_hive").show()


change it to spark.sql("desc formatted normal_orc_as_source_hive").show(), is it ok?

How to get the warning and verify it in code?

gatorsmile · 2018-01-05T13:46:02Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala

+      "desc formatted normal_orc_as_source_hive").show()
+    checkAnswer(sql("SELECT COUNT(*) FROM normal_orc_as_source_hive"), Row(10))
+    assert(HiveSerDe.sourceToSerDe("org.apache.spark.sql.hive.orc")
+      .equals(HiveSerDe.sourceToSerDe("orc")))


Also add the checks:

assert(HiveSerDe.sourceToSerDe("org.apache.spark.sql.orc") .equals(HiveSerDe.sourceToSerDe("orc")))

SparkQA · 2018-01-05T13:59:32Z

Test build #85724 has finished for PR 20165 at commit cf7cbce.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2018-01-05T15:19:48Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala

+         |  PATH '${new File(orcTableAsDir.getAbsolutePath).toURI}'
+         |)
+       """.stripMargin)
+    spark.sql("desc formatted normal_orc_as_source_hive").show()


Replace it by

val tableMetadata = spark.sessionState.catalog.getTableMetadata( TableIdentifier("normal_orc_as_source_hive")) assert(tableMetadata.storage.inputFormat == Option("org.apache.hadoop.hive.ql.io.orc.OrcInputFormat")) assert(tableMetadata.storage.outputFormat == Option("org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat")) assert(tableMetadata.storage.serde == Option("org.apache.hadoop.hive.ql.io.orc.OrcSerde"))

gatorsmile · 2018-01-05T15:20:06Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala

@@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends OrcSuite with TestHiveSingleton {
       """.stripMargin)
  }

+  test("SPARK-22972: hive orc source") {
+    spark.sql(


WithTable("normal_orc_as_source_hive")

gatorsmile · 2018-01-05T15:20:31Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala

@@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends OrcSuite with TestHiveSingleton {
       """.stripMargin)
  }

+  test("SPARK-22972: hive orc source") {
+    spark.sql(
+      s"""CREATE TABLE normal_orc_as_source_hive


Put it to next line. You can refer the other test cases of CREATE TABLE

SparkQA · 2018-01-05T15:27:02Z

Test build #85723 has finished for PR 20165 at commit fa902d6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-01-05T15:57:56Z

Test build #85725 has finished for PR 20165 at commit b64ce53.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-01-08T04:47:31Z

Test build #85778 has finished for PR 20165 at commit 1cfa72f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-01-08T04:58:12Z

Test build #85779 has finished for PR 20165 at commit cc4dd13.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2018-01-08T08:15:00Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala

+  test("SPARK-22972: hive orc source") {
+    val tableName = "normal_orc_as_source_hive"
+    withTable(tableName) {
+


Remove this line

gatorsmile · 2018-01-08T08:15:38Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala

+           |OPTIONS (
+           |  PATH '${new File(orcTableAsDir.getAbsolutePath).toURI}'
+           |)
+       """.stripMargin)


Nit:

s""" |CREATE TABLE $tableName |USING org.apache.spark.sql.hive.orc |OPTIONS ( | PATH '${new File(orcTableAsDir.getAbsolutePath).toURI}' |) """.stripMargin)

Be careful about the indents.

gatorsmile · 2018-01-08T08:16:15Z

LGTM except two trivial comments.

SparkQA · 2018-01-08T16:11:49Z

Test build #85796 has finished for PR 20165 at commit 3eafdfb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…provider org.apache.spark.sql.hive.orc ## What changes were proposed in this pull request? Fix the warning: Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc. ## How was this patch tested? test("SPARK-22972: hive orc source") assert(HiveSerDe.sourceToSerDe("org.apache.spark.sql.hive.orc") .equals(HiveSerDe.sourceToSerDe("orc"))) Author: xubo245 <[email protected]> Closes #20165 from xubo245/HiveSerDe. (cherry picked from commit 68ce792) Signed-off-by: gatorsmile <[email protected]>

gatorsmile · 2018-01-09T02:18:25Z

Thanks! Merged to master/2.3

Could you submit a separate PR for 2.2?

xubo245 · 2018-01-09T03:29:09Z

Thank you too.
Ok, I will raise a PR for this later.

…provider org.apache.spark.sql.hive.orc Fix the warning: Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc. For branch-2.2, it cherry-pick from apache@8032cf8 test("SPARK-22972: hive orc source") assert(HiveSerDe.sourceToSerDe("org.apache.spark.sql.hive.orc") .equals(HiveSerDe.sourceToSerDe("orc"))) Author: xubo245 <[email protected]> Closes apache#20165 from xubo245/HiveSerDe.

…provider org.apache.spark.sql.hive.orc ## What changes were proposed in this pull request? Fix the warning: Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc. This PR is for branch-2.2 and cherry-pick from 8032cf8 The old PR is #20165 ## How was this patch tested? Please see test("SPARK-22972: hive orc source") Author: xubo245 <[email protected]> Closes #20195 from xubo245/HiveSerDeForBranch2.2.

…provider org.apache.spark.sql.hive.orc ## What changes were proposed in this pull request? Fix the warning: Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc. This PR is for branch-2.2 and cherry-pick from apache@8032cf8 The old PR is apache#20165 ## How was this patch tested? Please see test("SPARK-22972: hive orc source") Author: xubo245 <[email protected]> Closes apache#20195 from xubo245/HiveSerDeForBranch2.2.

[SPARK-22972] Couldn't find corresponding Hive SerDe for data source …

fa902d6

…provider org.apache.spark.sql.hive.orc.

gatorsmile reviewed Jan 5, 2018

View reviewed changes

keep the original one and update code style

cf7cbce

fix scala style: Whitespace at end of line

b64ce53

gatorsmile reviewed Jan 5, 2018

View reviewed changes

xubo245 added 2 commits January 8, 2018 09:40

add withTable and tableSchema

1cfa72f

add a line

cc4dd13

gatorsmile reviewed Jan 8, 2018

View reviewed changes

update style

3eafdfb

asfgit closed this in 68ce792 Jan 9, 2018

xubo245 mentioned this pull request Jan 9, 2018

[SPARK-22972][SQL] Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc #20195

Closed

cloud-fan mentioned this pull request Jan 8, 2019

[SPARK-26571][SQL] Update Hive Serde mapping with canonical name of Parquet and Orc FileFormat #23491

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-22972] Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc #20165

[SPARK-22972] Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc #20165

xubo245 commented Jan 5, 2018 •

edited

Loading

gatorsmile commented Jan 5, 2018

gatorsmile Jan 5, 2018

xubo245 Jan 5, 2018

gatorsmile Jan 5, 2018

xubo245 Jan 5, 2018

gatorsmile Jan 5, 2018

xubo245 Jan 5, 2018

gatorsmile Jan 5, 2018

xubo245 Jan 5, 2018

SparkQA commented Jan 5, 2018

gatorsmile Jan 5, 2018

xubo245 Jan 8, 2018

gatorsmile Jan 5, 2018

xubo245 Jan 8, 2018

gatorsmile Jan 5, 2018 •

edited

Loading

xubo245 Jan 8, 2018

SparkQA commented Jan 5, 2018

SparkQA commented Jan 5, 2018

SparkQA commented Jan 8, 2018

SparkQA commented Jan 8, 2018

gatorsmile Jan 8, 2018

xubo245 Jan 8, 2018

gatorsmile Jan 8, 2018

gatorsmile Jan 8, 2018

xubo245 Jan 8, 2018

gatorsmile commented Jan 8, 2018

SparkQA commented Jan 8, 2018

gatorsmile commented Jan 9, 2018

xubo245 commented Jan 9, 2018

[SPARK-22972] Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc #20165

[SPARK-22972] Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc #20165

Conversation

xubo245 commented Jan 5, 2018 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

gatorsmile commented Jan 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jan 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gatorsmile Jan 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jan 5, 2018

SparkQA commented Jan 5, 2018

SparkQA commented Jan 8, 2018

SparkQA commented Jan 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gatorsmile commented Jan 8, 2018

SparkQA commented Jan 8, 2018

gatorsmile commented Jan 9, 2018

xubo245 commented Jan 9, 2018

xubo245 commented Jan 5, 2018 •

edited

Loading

gatorsmile Jan 5, 2018 •

edited

Loading