[SPARK-3826][SQL]enable hive-thriftserver to support hive-0.13.1 #2685

scwf · 2014-10-07T01:49:43Z

In #2241 hive-thriftserver is not enabled. This patch enable hive-thriftserver to support hive-0.13.1 by using a shim layer refer to #2241.

1 A light shim layer(code in sql/hive-thriftserver/hive-version) for each different hive version to handle api compatibility

2 New pom profiles "hive-default" and "hive-versions"(copy from #2241) to activate different hive version

3 SBT cmd for different version as follows:
hive-0.12.0 --- sbt/sbt -Phive,hadoop-2.3 -Phive-0.12.0 assembly
hive-0.13.1 --- sbt/sbt -Phive,hadoop-2.3 -Phive-0.13.1 assembly

4 Since hive-thriftserver depend on hive subproject, this patch should be merged with #2241 to enable hive-0.13.1 for hive-thriftserver

AmplabJenkins · 2014-10-07T01:52:10Z

Can one of the admins verify this patch?

SparkQA · 2014-10-07T04:00:11Z

QA tests have started for PR 2685 at commit 3a08b14.

This patch merges cleanly.

SparkQA · 2014-10-07T04:01:13Z

QA tests have finished for PR 2685 at commit 3a08b14.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-09T20:32:24Z

Can one of the admins verify this patch?

scwf · 2014-10-13T07:39:27Z

@pwendell, i am resolving the conflicts, other TODO's here?

…riftserver

zhzhan · 2014-10-22T23:03:49Z

pom.xml

+        <activeByDefault>false</activeByDefault>
+      </activation>
+      <modules>
+        <module>sql/hive-thriftserver</module>


After hive-0.13.1 is committed, sql/hive-thirftserver can be put to top level instead of here.

you mean move to <modules> upper in this pom?

Yes. either move to modules, or move to hive profile, since it is supported in both versions.

…riftserver1

marmbrus · 2014-10-26T21:27:46Z

Now that Hive 13 is merged in it would be great to get this in ASAP. I looked over this and it seems pretty good. My only high level comment is maybe we should keep all the Hive Shim code in a single project instead of having version specific code in both hive and hive-thrift server. That would simplify the build and consolidate the places where we have these hacks. It woudl also allow us to avoid duplicating things like getCommandProcessor in both Shims. Thoughts?

@pwendell can you glance over the (limited) build changes.
@liancheng can you look this over as well?

pwendell · 2014-10-26T22:08:32Z

dev/run-tests

@@ -167,7 +167,7 @@ CURRENT_BLOCK=$BLOCK_SPARK_UNIT_TESTS
  # If the Spark SQL tests are enabled, run the tests with the Hive profiles enabled.
  # This must be a single argument, as it is.
  if [ -n "$_RUN_SQL_TESTS" ]; then
-    SBT_MAVEN_PROFILES_ARGS="$SBT_MAVEN_PROFILES_ARGS -Phive"
+    SBT_MAVEN_PROFILES_ARGS="$SBT_MAVEN_PROFILES_ARGS -Phive -Phive-0.12.0"


is this change ncessary? seems like it might be good to leave it how it is now

Yeah agreed. If Hive13 is the default we should test Hive13.

pwendell · 2014-10-26T22:12:03Z

Some minor comments on the build stuff. I think we're close, it's just small stuff.

The thirift server is not available in the default (hive13) profile yet which is breaking all SQL only PRs. This turns off these test until #2685 is merged. Author: Michael Armbrust <[email protected]> Closes #2950 from marmbrus/fixTests and squashes the following commits: 1a6dfee [Michael Armbrust] [HOTFIX][SQL] Temporarily turn of hive-server tests.

scwf · 2014-10-27T03:43:19Z

@marmbrus, how about make a new sub project named hive-shim to keep all the Hive Shim code in it?

AmplabJenkins · 2014-10-31T05:52:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22591/
Test PASSed.

liancheng · 2014-10-31T05:57:10Z

Yay! It passed!

scwf · 2014-10-31T05:59:12Z

So will you re-publish the hive 0.13 jar @pwendell? or use 0.13.1a?

pwendell · 2014-10-31T06:01:48Z

it is not possible to mutate them after the fact. Let's stick with 0.13.1a - I will fully release it now. But don't remove the extra repository (we can remove it later) because it takes some time to propagate.

pwendell · 2014-10-31T06:04:00Z

dev/run-tests

  echo -e "q\n" \
-    | sbt/sbt $BUILD_MVN_PROFILE_ARGS clean package assembly/assembly \
+    | sbt/sbt $BUILD_MVN_PROFILE_ARGS clean hive/compile hive-thriftserver/compile \


I actually don't think this is compiling against Hive 0.12 right now... is it?

i think it's against 0.12 because BUILD_MVN_PROFILE_ARGS="$SBT_MAVEN_PROFILES_ARGS -Phive -Phive-0.12.0", right?

It should be, BUILD_MVN_PROFILE_ARGS is defined above with -Phive-0.12.0:

BUILD_MVN_PROFILE_ARGS="$SBT_MAVEN_PROFILES_ARGS -Phive -Phive-0.12.0"

zhzhan · 2014-10-31T06:15:01Z

whew, finally.

SparkQA · 2014-10-31T06:45:07Z

Test build #22602 has started for PR 2685 at commit f26f3be.

This patch merges cleanly.

liancheng · 2014-10-31T07:27:16Z

@zhzhan Had a glance at the Kryo issue you pointed out, it should be related to the POM inconsistency problem I mentioned above, but I'm not sure whether they are identical since this issue doesn't mention any specific Kryo version. I'll investigate this later. Thanks for pointing out this issue!

Although Jenkins finally nods, I'm still puzzled with the root cause of the original build failure. We only know that introducing Kryo 2.22 prevents un-shaded Objenesis classes from being included in the assembly jar and thus breaks the core tests, but how and why? Also, is it 100% safe to downgrade Kryo 2.22 to Kryo 2.21 for Hive 0.13.1?

SparkQA · 2014-10-31T08:29:51Z

Test build #22602 has finished for PR 2685 at commit f26f3be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-31T08:29:56Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22602/
Test PASSed.

marmbrus · 2014-10-31T18:27:30Z

Thanks guys for all your hard work on this! Merging to master.

coderfi · 2014-11-02T18:19:51Z

In /pom.xml

<properties>
    <hive.version>0.13.1a</hive.version>
</properties>
...
<profile>
  <id>hive-0.13.1</id>
  ...
  <properties>
    <hive.version>0.13.1</hive.version>
    ...
  </properties>
</profile>

The 'hive-0.13.1' is overriding the hive.version back to the regular 'hive-0.13.1' version, instead of referencing 'hive-0.13.1a'.

It seems mvn -Phive would get me a build with hive-0.13a containing this pull's patches.

However, I was building with mvn -Phive -Phive-0.13.1, so I ran into the 'NoClassDefFoundError ... InstantiatorStrategy' issue.
Omitting the 'hive-0.13.1' profile got me past the issue (since the pom defaults effectively gives me hive 0.13(a) anyway, which is what I want).

liancheng · 2014-11-03T03:24:44Z

@coderfi This is a good catch, would you mind to file a JIRA ticket for this? A PR would be even better :)

scwf · 2014-11-03T04:05:33Z

Yes, here should be 0.13.1a.

pwendell · 2014-11-03T05:57:15Z

Yes - we just need to change that to 0.13.1a

On Sun, Nov 2, 2014 at 8:05 PM, wangfei [email protected] wrote:

Yes, here should be 0.13.1a.

—
Reply to this email directly or view it on GitHub
#2685 (comment).

scwf · 2014-11-03T06:17:19Z

And note just to change hive.version to 0.13.1a, hive.version.short should be 0.13.1.

coderfi · 2014-11-03T08:23:12Z

@liancheng PR #3072 created (all of one line! :) ).

instead of `hive.version=0.13.1`. e.g. mvn -Phive -Phive=0.13.1 Note: `hive.version=0.13.1a` is the default property value. However, when explicitly specifying the `hive-0.13.1` maven profile, the wrong one would be selected. References: PR #2685, which resolved a package incompatibility issue with Hive-0.13.1 by introducing a special version Hive-0.13.1a Author: fi <[email protected]> Closes #3072 from coderfi/master and squashes the following commits: 7ca4b1e [fi] Fixes the `hive-0.13.1` maven profile referencing `hive.version=0.13.1` instead of the Spark compatible `hive.version=0.13.1a` Note: `hive.version=0.13.1a` is the default version. However, when explicitly specifying the `hive-0.13.1` maven profile, the wrong one would be selected. e.g. mvn -Phive -Phive=0.13.1 See PR #2685 (cherry picked from commit df607da) Signed-off-by: Michael Armbrust <[email protected]>

instead of `hive.version=0.13.1`. e.g. mvn -Phive -Phive=0.13.1 Note: `hive.version=0.13.1a` is the default property value. However, when explicitly specifying the `hive-0.13.1` maven profile, the wrong one would be selected. References: PR #2685, which resolved a package incompatibility issue with Hive-0.13.1 by introducing a special version Hive-0.13.1a Author: fi <[email protected]> Closes #3072 from coderfi/master and squashes the following commits: 7ca4b1e [fi] Fixes the `hive-0.13.1` maven profile referencing `hive.version=0.13.1` instead of the Spark compatible `hive.version=0.13.1a` Note: `hive.version=0.13.1a` is the default version. However, when explicitly specifying the `hive-0.13.1` maven profile, the wrong one would be selected. e.g. mvn -Phive -Phive=0.13.1 See PR #2685

scwf mentioned this pull request Oct 13, 2014

[SPARK-2706][SQL] Enable Spark to support Hive 0.13 #2241

Closed

scwf added 3 commits October 13, 2014 22:01

update and fix conflicts

ae47489

update pom according spark-2706

7c66b8e

Merge branch 'master' of https://github.com/apache/spark into shim-th…

c6da3ce

…riftserver

scwf force-pushed the shim-thriftserver1 branch from 6cfcec8 to c6da3ce Compare October 14, 2014 16:48

scwf added 4 commits October 14, 2014 09:53

update run-tests to run hive-0.12.0 default now

dfd1c63

fixed when result filed is null

0bc53aa

enable thriftserver in profile hive-0.13.1

4b681f4

fix small bug

13afde0

zhzhan reviewed Oct 22, 2014
View reviewed changes

scwf added 2 commits October 22, 2014 19:25

revert pom changes

41f727b

Merge branch 'master' of https://github.com/apache/spark into shim-th…

f48d3a5

…riftserver1

scwf force-pushed the shim-thriftserver1 branch from 9e5e2fc to f48d3a5 Compare October 23, 2014 02:45

update and fix conflicts

f51ff4e

marmbrus mentioned this pull request Oct 26, 2014

[HOTFIX][SQL] Temporarily turn of hive-server tests. #2950

Closed

pwendell reviewed Oct 26, 2014
View reviewed changes

scwf added 2 commits October 26, 2014 20:07

move hive module to hive profile

3529e98

sql/hive included since examples depend on it

52674a4

pwendell reviewed Oct 31, 2014
View reviewed changes

remove clean to save time

f26f3be

liancheng mentioned this pull request Oct 31, 2014

[SPARK-3791][SQL] Provides Spark version and Hive version in HiveThriftServer2 #2843

Closed

1 task

asfgit closed this in 7c41d13 Oct 31, 2014

scwf deleted the shim-thriftserver1 branch November 3, 2014 04:04

coderfi mentioned this pull request Nov 3, 2014

[SPARK-4211][Build] Fixes hive.version in Maven profile hive-0.13.1 #3072

Closed

liancheng mentioned this pull request Nov 14, 2014

[SPARK-4309][SPARK-4407][SQL] Date type support for Thrift server, and fixes for complex types #3178

Closed

adrian-wang mentioned this pull request Nov 25, 2014

[SPARK-4599] [Build] [SQL] add hive profile in root pom #3450

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-3826][SQL]enable hive-thriftserver to support hive-0.13.1 #2685

[SPARK-3826][SQL]enable hive-thriftserver to support hive-0.13.1 #2685

scwf commented Oct 7, 2014

AmplabJenkins commented Oct 7, 2014

SparkQA commented Oct 7, 2014

SparkQA commented Oct 7, 2014

AmplabJenkins commented Oct 9, 2014

scwf commented Oct 13, 2014

zhzhan Oct 22, 2014

scwf Oct 22, 2014

zhzhan Oct 23, 2014

marmbrus commented Oct 26, 2014

pwendell Oct 26, 2014

marmbrus Oct 26, 2014

pwendell commented Oct 26, 2014

scwf commented Oct 27, 2014

AmplabJenkins commented Oct 31, 2014

liancheng commented Oct 31, 2014

scwf commented Oct 31, 2014

pwendell commented Oct 31, 2014

pwendell Oct 31, 2014

scwf Oct 31, 2014

liancheng Oct 31, 2014

zhzhan commented Oct 31, 2014

SparkQA commented Oct 31, 2014

liancheng commented Oct 31, 2014

SparkQA commented Oct 31, 2014

AmplabJenkins commented Oct 31, 2014

marmbrus commented Oct 31, 2014

coderfi commented Nov 2, 2014

liancheng commented Nov 3, 2014

scwf commented Nov 3, 2014

pwendell commented Nov 3, 2014

scwf commented Nov 3, 2014

coderfi commented Nov 3, 2014

[SPARK-3826][SQL]enable hive-thriftserver to support hive-0.13.1 #2685

[SPARK-3826][SQL]enable hive-thriftserver to support hive-0.13.1 #2685

Conversation

scwf commented Oct 7, 2014

AmplabJenkins commented Oct 7, 2014

SparkQA commented Oct 7, 2014

SparkQA commented Oct 7, 2014

AmplabJenkins commented Oct 9, 2014

scwf commented Oct 13, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marmbrus commented Oct 26, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pwendell commented Oct 26, 2014

scwf commented Oct 27, 2014

AmplabJenkins commented Oct 31, 2014

liancheng commented Oct 31, 2014

scwf commented Oct 31, 2014

pwendell commented Oct 31, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhzhan commented Oct 31, 2014

SparkQA commented Oct 31, 2014

liancheng commented Oct 31, 2014

SparkQA commented Oct 31, 2014

AmplabJenkins commented Oct 31, 2014

marmbrus commented Oct 31, 2014

coderfi commented Nov 2, 2014

liancheng commented Nov 3, 2014

scwf commented Nov 3, 2014

pwendell commented Nov 3, 2014

scwf commented Nov 3, 2014

coderfi commented Nov 3, 2014