-
Notifications
You must be signed in to change notification settings - Fork 28.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improved build configuration #480
Conversation
Can one of the admins verify this patch? |
@@ -793,6 +831,157 @@ | |||
</build> | |||
|
|||
<profiles> | |||
<!-- SPARK-1121: Adds an explicit dependency on Avro to work around a Hadoop 0.23.X issue --> | |||
<profile> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this copy-and-paste approach is reasonable to maintain. Have you tried writing a rule that triggers on a version range of the hadoop-client
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do not know how to do, an example?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Scratch that. I tried a bunch of variations on this and can't find a way to activate the profile based on a range. I think it's best to activate this profile explicitly with -P
rather than maintain all this duplication, still.
@@ -892,10 +1081,11 @@ | |||
<dependency> | |||
<groupId>org.apache.zookeeper</groupId> | |||
<artifactId>zookeeper</artifactId> | |||
<version>3.4.5</version> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, this looks like it was unspecified. Does this version need to vary with Hadoop version? And also in the SBT build?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
curator-recipes 2.4.0 => zookeeper 3.4.5
hbase 0.94.6 => zookeeper 3.4.5
kafka_2.10 0.8.0 => zookeeper 3.3.4
hadoop is not directly depend on the zookeeper
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see, this declaration is just there to make zookeeper "provided" in order to not conflict with a cluster's copy/version of ZK from Hadoop. Hadoop version kind of matters in that sense. I wonder if this is actually best left unspecified then, since there is no intent to enforce any particular version of ZK, just make sure it is not in the artifact when built with this profile?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SPARK-1064,PR 102.There is no equivalent feature in sbt.
@@ -793,6 +833,17 @@ | |||
</build> | |||
|
|||
<profiles> | |||
<!-- SPARK-1121: Adds an explicit dependency on Avro to work around a Hadoop 0.23.X issue --> | |||
<profile> | |||
<id>hadoop-0.23</id> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think in the past we had trouble defining this profile in the parent pom. The issue was that profiles were not correctly inherited by the sub projects, because profiles are not transative. I don't remember how exactly this manifested though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have not found this problem in the test.
@@ -39,6 +39,9 @@ For Apache Hadoop versions 1.x, Cloudera CDH MRv1, and other Hadoop versions wit | |||
# Cloudera CDH 4.2.0 with MapReduce v1 | |||
$ mvn -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -DskipTests clean package | |||
|
|||
# Apache Hadoop 0.23.x | |||
$ mvn -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean package | |||
|
|||
For Apache Hadoop 2.x, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions with YARN, you should enable the "yarn-alpha" or "yarn" profile and set the "hadoop.version", "yarn.version" property: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
now that we support building without YARN... this should say "you can enable" instead of "you should enable"
Jenkins, test this please. |
Merged build triggered. |
Merged build started. |
So I looked through this a bunch and ran several builds with different permutations. This looks really good! Thanks @witgo I'm sure it took some time to figure all this out. I'm going to merge this pending tests because I'd like to cut an RC tonight and I don't want to change the build around after we ask people to test stuff. |
Merged build finished. All automated tests passed. |
All automated tests passed. |
I've merged this, thanks. |
1, Fix SPARK-1441: compile spark core error with hadoop 0.23.x 2, Fix SPARK-1491: maven hadoop-provided profile fails to build 3, Fix org.scala-lang: * ,org.apache.avro:* inconsistent versions dependency 4, A modified on the sql/catalyst/pom.xml,sql/hive/pom.xml,sql/core/pom.xml (Four spaces formatted into two spaces) Author: witgo <[email protected]> Closes #480 from witgo/format_pom and squashes the following commits: 03f652f [witgo] review commit b452680 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom bee920d [witgo] revert fix SPARK-1629: Spark Core missing commons-lang dependence 7382a07 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 6902c91 [witgo] fix SPARK-1629: Spark Core missing commons-lang dependence 0da4bc3 [witgo] merge master d1718ed [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom e345919 [witgo] add avro dependency to yarn-alpha 77fad08 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 62d0862 [witgo] Fix org.scala-lang: * inconsistent versions dependency 1a162d7 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 934f24d [witgo] review commit cf46edc [witgo] exclude jruby 06e7328 [witgo] Merge branch 'SparkBuild' into format_pom 99464d2 [witgo] fix maven hadoop-provided profile fails to build 0c6c1fc [witgo] Fix compile spark core error with hadoop 0.23.x 6851bec [witgo] Maintain consistent SparkBuild.scala, pom.xml (cherry picked from commit 030f2c2) Conflicts: sql/catalyst/pom.xml sql/core/pom.xml sql/hive/pom.xml
Cool! |
Handful of 0.9 fixes This patch addresses a few fixes for Spark 0.9.0 based on the last release candidate. @mridulm gets credit for reporting most of the issues here. Many of the fixes here are based on his work in apache#477 and follow up discussion with him.
1, Fix SPARK-1441: compile spark core error with hadoop 0.23.x 2, Fix SPARK-1491: maven hadoop-provided profile fails to build 3, Fix org.scala-lang: * ,org.apache.avro:* inconsistent versions dependency 4, A modified on the sql/catalyst/pom.xml,sql/hive/pom.xml,sql/core/pom.xml (Four spaces formatted into two spaces) Author: witgo <[email protected]> Closes apache#480 from witgo/format_pom and squashes the following commits: 03f652f [witgo] review commit b452680 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom bee920d [witgo] revert fix SPARK-1629: Spark Core missing commons-lang dependence 7382a07 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 6902c91 [witgo] fix SPARK-1629: Spark Core missing commons-lang dependence 0da4bc3 [witgo] merge master d1718ed [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom e345919 [witgo] add avro dependency to yarn-alpha 77fad08 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 62d0862 [witgo] Fix org.scala-lang: * inconsistent versions dependency 1a162d7 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 934f24d [witgo] review commit cf46edc [witgo] exclude jruby 06e7328 [witgo] Merge branch 'SparkBuild' into format_pom 99464d2 [witgo] fix maven hadoop-provided profile fails to build 0c6c1fc [witgo] Fix compile spark core error with hadoop 0.23.x 6851bec [witgo] Maintain consistent SparkBuild.scala, pom.xml
Handful of 0.9 fixes This patch addresses a few fixes for Spark 0.9.0 based on the last release candidate. @mridulm gets credit for reporting most of the issues here. Many of the fixes here are based on his work in apache#477 and follow up discussion with him. (cherry picked from commit 77b986f) Signed-off-by: Patrick Wendell <[email protected]>
###### _excavator_ is a bot for automating changes across repositories. Changes produced by the roomba/latest-gradle-wrapper-oss check. {runtimeCheckDesc} To enable or disable this check, please contact the maintainers of Excavator.
This patch fixes the issue mentioned in PR#1519. The default size of devstack swift is 2G, if we enable octavia or we create some new images into glance during test. We will get failure. The key reason for this is the swift device size is too small. So this patch will increase the size through pass a env parameter in devstack local conf. Close: theopenlab/openlab#222
… incorrect result (apache#475) (apache#480)
1, Fix SPARK-1441: compile spark core error with hadoop 0.23.x
2, Fix SPARK-1491: maven hadoop-provided profile fails to build
3, Fix org.scala-lang: * ,org.apache.avro:* inconsistent versions dependency
4, A modified on the sql/catalyst/pom.xml,sql/hive/pom.xml,sql/core/pom.xml (Four spaces formatted into two spaces)