[SPARK-20728][SQL] Make ORCFileFormat configurable between sql/hive and sql/core #17980

dongjoon-hyun · 2017-05-15T02:44:57Z

What changes were proposed in this pull request?

SPARK-20682 is trying to improve Apache Spark to have a new ORCFileFormat based on Apache ORC for many reasons.

On top of that, this PR depends on SPARK-20682 and aims to provide a configuration to choose the default ORCFileFormat from legacy sql/hive module or new sql/core module.

For example, this configuration will affects the following operations.

spark.read.orc(...)

CREATE TABLE t
USING ORC
...

Since SPARK-20682 (#17924 and #17943) are still under review, I'm inevitably including the dependent code. I'll update this and previous PR according to the review result.

How was this patch tested?

Pass the Jenkins with new test suites.

SparkQA · 2017-05-15T05:02:48Z

Test build #76924 has finished for PR 17980 at commit 7716234.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-15T05:42:35Z

Test build #76931 has started for PR 17980 at commit 73d56f2.

dongjoon-hyun · 2017-05-15T05:54:01Z

sql/core/src/test/scala/org/apache/spark/sql/sources/DDLSourceLoadSuite.scala

@@ -55,10 +56,12 @@ class DDLSourceLoadSuite extends DataSourceTest with SharedSQLContext {
  }

  test("should fail to load ORC without Hive Support") {


We can remove this test case when we remove sql/hive ORCFileFormat.

dongjoon-hyun · 2017-05-15T07:07:50Z

Retest this please.

SparkQA · 2017-05-15T10:04:03Z

Test build #76935 has finished for PR 17980 at commit 73d56f2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-17T06:15:09Z

Test build #77011 has started for PR 17980 at commit 87dd161.

dongjoon-hyun · 2017-05-17T15:33:00Z

Retest this please

SparkQA · 2017-05-17T19:57:22Z

Test build #77019 has finished for PR 17980 at commit 87dd161.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-24T00:31:58Z

Test build #77264 has finished for PR 17980 at commit 05716c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-06-10T21:53:11Z

Hi, @sameeragarwal , @cloud-fan , @gatorsmile .
Could you give me some advice on this? I think this is the start of Feature Parity for ORC.

dongjoon-hyun · 2017-06-16T21:35:12Z

Retest this please.

SparkQA · 2017-06-17T00:35:30Z

Test build #78192 has finished for PR 17980 at commit 05716c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-06T07:04:56Z

Test build #79264 has finished for PR 17980 at commit 39c3851.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-07-06T07:31:28Z

Retest this please.

SparkQA · 2017-07-06T10:28:58Z

Test build #79269 has finished for PR 17980 at commit 39c3851.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-07-06T22:46:46Z

Hi, @rxin, @sameeragarwal , @cloud-fan , @gatorsmile , @viirya .
Since Apache Spark 2.2.0 release is going to coming soon, it's a good time to restart this.
Could you give me some advice how to make ORC support up-to-date in Apache Spark 2.3?

SparkQA · 2017-07-09T12:02:35Z

Test build #79412 has finished for PR 17980 at commit cbc5bc2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-12T23:08:49Z

Test build #79570 has finished for PR 17980 at commit 5db6acf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2017-07-14T18:51:45Z

In general, this PR seems to have the large number of lines.
Are there any ways to split this PR into multiple smaller PRs?

dongjoon-hyun · 2017-07-14T21:31:03Z

Thank you for review, @kiszk .

The first one about adding new ORC source is more smaller than this .
Also, there is more smaller version without vectorized part.

If we need to split more, we can split the followings.

Adding Apache ORC dependency (pom and dependency changes)
OrcReadBenchmark (388 lines)

Is that helpful for review?

In general, 50% lines are new ORC test codes which is duplicated from old ORC test codes. Could you recommend some better way?

dongjoon-hyun · 2017-07-14T21:55:23Z

Hi, @kiszk . I will start with Adding Apache ORC dependency (pom and dependency changes) in
SPARK-21422 first.

kiszk · 2017-07-15T03:59:58Z

If new test cases work for the existing orc component, how about updating test cases at first?

dongjoon-hyun · 2017-07-15T15:03:33Z

I developed new ORC component and tests in sql/core module without sql/hive module. Do you mean add a sql/hive module test-dependency into sql module?

kiszk · 2017-07-17T14:37:00Z

I see. I thought that new tests are for old and new ORC components.

dongjoon-hyun · 2017-07-17T15:46:42Z

I'm trying to split the PR as you recommended before. Thank you always.

dongjoon-hyun · 2017-08-03T23:18:06Z

Retest this please.

SparkQA · 2017-08-04T02:25:01Z

Test build #80224 has finished for PR 17980 at commit 5db6acf.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-08-04T16:21:51Z

Retest this please

SparkQA · 2017-08-04T19:34:03Z

Test build #80254 has finished for PR 17980 at commit 5db6acf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-14T05:48:23Z

Test build #80603 has finished for PR 17980 at commit 5db6acf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

## What changes were proposed in this pull request? Like Parquet, this PR aims to depend on the latest Apache ORC 1.4 for Apache Spark 2.3. There are key benefits for Apache ORC 1.4. - Stability: Apache ORC 1.4.0 has many fixes and we can depend on ORC community more. - Maintainability: Reduce the Hive dependency and can remove old legacy code later. Later, we can get the following two key benefits by adding new ORCFileFormat in SPARK-20728 (apache#17980), too. - Usability: User can use ORC data sources without hive module, i.e, -Phive. - Speed: Use both Spark ColumnarBatch and ORC RowBatch together. This will be faster than the current implementation in Spark. ## How was this patch tested? Pass the jenkins. Author: Dongjoon Hyun <[email protected]> Closes apache#18640 from dongjoon-hyun/SPARK-21422.

SparkQA · 2017-08-16T07:59:21Z

Test build #80727 has finished for PR 17980 at commit 5a9958d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-16T10:09:22Z

Test build #80729 has finished for PR 17980 at commit 61b27e5.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-08-16T15:40:56Z

Retest this please.

SparkQA · 2017-08-16T18:19:58Z

Test build #80741 has finished for PR 17980 at commit 61b27e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-08-24T15:26:07Z

Retest this please.

SparkQA · 2017-08-24T15:33:01Z

Test build #81090 has finished for PR 17980 at commit 61b27e5.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-24T17:03:12Z

Test build #81095 has finished for PR 17980 at commit 02371c6.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-24T20:33:06Z

Test build #81096 has finished for PR 17980 at commit 04b8494.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-28T23:30:31Z

Test build #81190 has finished for PR 17980 at commit 257e5a6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-31T06:50:59Z

Test build #81274 has finished for PR 17980 at commit edead5d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-09-04T10:24:54Z

Test build #81379 has finished for PR 17980 at commit c50e997.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…nd sql/core

SparkQA · 2017-09-11T00:17:39Z

Test build #81612 has finished for PR 17980 at commit 01868ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2017-11-06T09:06:07Z

@dongjoon-hyun Do we still need this PR?

dongjoon-hyun · 2017-11-06T21:59:29Z

Thank you for review, @jiangxb1987 . Yes. This is still required. I'll rebase this PR after #19651 .

dongjoon-hyun · 2017-12-21T17:12:05Z

Almost codes inside this PR are merged now except #19943 .

dongjoon-hyun commented May 15, 2017

View reviewed changes

dongjoon-hyun mentioned this pull request Jul 14, 2017

[SPARK-21422][BUILD] Depend on Apache ORC 1.4.0 #18640

Closed

dongjoon-hyun mentioned this pull request Aug 16, 2017

[SPARK-20682][SQL] Update ORC data source based on Apache ORC library #18953

Closed

dongjoon-hyun mentioned this pull request Sep 3, 2017

[SPARK-20682][SQL] Support a new faster ORC data source based on Apache ORC #17924

Closed

This was referenced Sep 8, 2017

[SPARK-20682][SQL] Implement new ORC data source based on Apache ORC #17943

Closed

[SPARK-21791][SQL] ORC should support column names with dot #19004

Closed

[SPARK-20728][SQL] Make ORCFileFormat configurable between sql/hive a…

01868ab

…nd sql/core

dongjoon-hyun mentioned this pull request Oct 26, 2017

[SPARK-15474][SQL] Write and read back non-emtpy schema with empty dataframe #19571

Closed

dongjoon-hyun mentioned this pull request Nov 3, 2017

[SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileFormat based on ORC 1.4.1 #19651

Closed

dongjoon-hyun mentioned this pull request Dec 4, 2017

[SPARK-20728][SQL] Make OrcFileFormat configurable between sql/hive and sql/core #19871

Closed

dongjoon-hyun closed this Dec 21, 2017

dongjoon-hyun deleted the SPARK-20728 branch December 21, 2017 17:12

		@@ -55,10 +56,12 @@ class DDLSourceLoadSuite extends DataSourceTest with SharedSQLContext {
		}

		test("should fail to load ORC without Hive Support") {

[SPARK-20728][SQL] Make ORCFileFormat configurable between sql/hive and sql/core #17980

[SPARK-20728][SQL] Make ORCFileFormat configurable between sql/hive and sql/core #17980

Conversation

dongjoon-hyun commented May 15, 2017 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented May 15, 2017

SparkQA commented May 15, 2017

dongjoon-hyun May 15, 2017

Choose a reason for hiding this comment

dongjoon-hyun commented May 15, 2017

SparkQA commented May 15, 2017

SparkQA commented May 17, 2017

dongjoon-hyun commented May 17, 2017

SparkQA commented May 17, 2017

SparkQA commented May 24, 2017

dongjoon-hyun commented Jun 10, 2017

dongjoon-hyun commented Jun 16, 2017

SparkQA commented Jun 17, 2017

SparkQA commented Jul 6, 2017

dongjoon-hyun commented Jul 6, 2017

SparkQA commented Jul 6, 2017

dongjoon-hyun commented Jul 6, 2017

SparkQA commented Jul 9, 2017

SparkQA commented Jul 12, 2017

kiszk commented Jul 14, 2017

dongjoon-hyun commented Jul 14, 2017 • edited Loading

dongjoon-hyun commented Jul 14, 2017

kiszk commented Jul 15, 2017 • edited Loading

dongjoon-hyun commented Jul 15, 2017

kiszk commented Jul 17, 2017

dongjoon-hyun commented Jul 17, 2017 • edited Loading

dongjoon-hyun commented Aug 3, 2017

SparkQA commented Aug 4, 2017

dongjoon-hyun commented Aug 4, 2017

SparkQA commented Aug 4, 2017

SparkQA commented Aug 14, 2017

SparkQA commented Aug 16, 2017

SparkQA commented Aug 16, 2017

dongjoon-hyun commented Aug 16, 2017

SparkQA commented Aug 16, 2017

dongjoon-hyun commented Aug 24, 2017

SparkQA commented Aug 24, 2017

SparkQA commented Aug 24, 2017

SparkQA commented Aug 24, 2017

SparkQA commented Aug 28, 2017

SparkQA commented Aug 31, 2017

SparkQA commented Sep 4, 2017

SparkQA commented Sep 11, 2017

jiangxb1987 commented Nov 6, 2017

dongjoon-hyun commented Nov 6, 2017

dongjoon-hyun commented Dec 21, 2017

dongjoon-hyun commented May 15, 2017 •

edited

Loading

dongjoon-hyun commented Jul 14, 2017 •

edited

Loading

kiszk commented Jul 15, 2017 •

edited

Loading

dongjoon-hyun commented Jul 17, 2017 •

edited

Loading