Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved build configuration #480

Closed
wants to merge 17 commits into from
Closed

Improved build configuration #480

wants to merge 17 commits into from

Conversation

witgo
Copy link
Contributor

@witgo witgo commented Apr 22, 2014

1, Fix SPARK-1441: compile spark core error with hadoop 0.23.x
2, Fix SPARK-1491: maven hadoop-provided profile fails to build
3, Fix org.scala-lang: * ,org.apache.avro:* inconsistent versions dependency
4, A modified on the sql/catalyst/pom.xml,sql/hive/pom.xml,sql/core/pom.xml (Four spaces formatted into two spaces)

@witgo witgo changed the title Improved build configuration [WIP]Improved build configuration Apr 22, 2014
@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@@ -793,6 +831,157 @@
</build>

<profiles>
<!-- SPARK-1121: Adds an explicit dependency on Avro to work around a Hadoop 0.23.X issue -->
<profile>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this copy-and-paste approach is reasonable to maintain. Have you tried writing a rule that triggers on a version range of the hadoop-client?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not know how to do, an example?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scratch that. I tried a bunch of variations on this and can't find a way to activate the profile based on a range. I think it's best to activate this profile explicitly with -P rather than maintain all this duplication, still.

@@ -892,10 +1081,11 @@
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.5</version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, this looks like it was unspecified. Does this version need to vary with Hadoop version? And also in the SBT build?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curator-recipes 2.4.0 => zookeeper 3.4.5
hbase 0.94.6 => zookeeper 3.4.5
kafka_2.10 0.8.0 => zookeeper 3.3.4
hadoop is not directly depend on the zookeeper

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, this declaration is just there to make zookeeper "provided" in order to not conflict with a cluster's copy/version of ZK from Hadoop. Hadoop version kind of matters in that sense. I wonder if this is actually best left unspecified then, since there is no intent to enforce any particular version of ZK, just make sure it is not in the artifact when built with this profile?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-1064,PR 102.There is no equivalent feature in sbt.

@witgo witgo changed the title [WIP]Improved build configuration Improved build configuration Apr 23, 2014
@@ -793,6 +833,17 @@
</build>

<profiles>
<!-- SPARK-1121: Adds an explicit dependency on Avro to work around a Hadoop 0.23.X issue -->
<profile>
<id>hadoop-0.23</id>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in the past we had trouble defining this profile in the parent pom. The issue was that profiles were not correctly inherited by the sub projects, because profiles are not transative. I don't remember how exactly this manifested though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not found this problem in the test.

@witgo witgo mentioned this pull request Apr 29, 2014
@@ -39,6 +39,9 @@ For Apache Hadoop versions 1.x, Cloudera CDH MRv1, and other Hadoop versions wit
# Cloudera CDH 4.2.0 with MapReduce v1
$ mvn -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -DskipTests clean package

# Apache Hadoop 0.23.x
$ mvn -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean package

For Apache Hadoop 2.x, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions with YARN, you should enable the "yarn-alpha" or "yarn" profile and set the "hadoop.version", "yarn.version" property:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now that we support building without YARN... this should say "you can enable" instead of "you should enable"

@pwendell
Copy link
Contributor

Jenkins, test this please.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@pwendell
Copy link
Contributor

So I looked through this a bunch and ran several builds with different permutations. This looks really good! Thanks @witgo I'm sure it took some time to figure all this out. I'm going to merge this pending tests because I'd like to cut an RC tonight and I don't want to change the build around after we ask people to test stuff.

@AmplabJenkins
Copy link

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14552/

@pwendell
Copy link
Contributor

I've merged this, thanks.

asfgit pushed a commit that referenced this pull request Apr 29, 2014
1, Fix SPARK-1441: compile spark core error with hadoop 0.23.x
2, Fix SPARK-1491: maven hadoop-provided profile fails to build
3, Fix org.scala-lang: * ,org.apache.avro:* inconsistent versions dependency
4, A modified on the sql/catalyst/pom.xml,sql/hive/pom.xml,sql/core/pom.xml (Four spaces formatted into two spaces)

Author: witgo <[email protected]>

Closes #480 from witgo/format_pom and squashes the following commits:

03f652f [witgo] review commit
b452680 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
bee920d [witgo] revert fix SPARK-1629: Spark Core missing commons-lang dependence
7382a07 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
6902c91 [witgo] fix SPARK-1629: Spark Core missing commons-lang dependence
0da4bc3 [witgo] merge master
d1718ed [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
e345919 [witgo] add avro dependency to yarn-alpha
77fad08 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
62d0862 [witgo] Fix org.scala-lang: * inconsistent versions dependency
1a162d7 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
934f24d [witgo] review commit
cf46edc [witgo] exclude jruby
06e7328 [witgo] Merge branch 'SparkBuild' into format_pom
99464d2 [witgo] fix maven hadoop-provided profile fails to build
0c6c1fc [witgo] Fix compile spark core error with hadoop 0.23.x
6851bec [witgo] Maintain consistent SparkBuild.scala, pom.xml
(cherry picked from commit 030f2c2)

Conflicts:

	sql/catalyst/pom.xml
	sql/core/pom.xml
	sql/hive/pom.xml
@asfgit asfgit closed this in 030f2c2 Apr 29, 2014
@witgo witgo deleted the format_pom branch April 29, 2014 06:46
@witgo
Copy link
Contributor Author

witgo commented Apr 29, 2014

Cool!

pwendell added a commit to pwendell/spark that referenced this pull request May 12, 2014
Handful of 0.9 fixes

This patch addresses a few fixes for Spark 0.9.0 based on the last release candidate.

@mridulm gets credit for reporting most of the issues here. Many of the fixes here are based on his work in apache#477 and follow up discussion with him.
pdeyhim pushed a commit to pdeyhim/spark-1 that referenced this pull request Jun 25, 2014
1, Fix SPARK-1441: compile spark core error with hadoop 0.23.x
2, Fix SPARK-1491: maven hadoop-provided profile fails to build
3, Fix org.scala-lang: * ,org.apache.avro:* inconsistent versions dependency
4, A modified on the sql/catalyst/pom.xml,sql/hive/pom.xml,sql/core/pom.xml (Four spaces formatted into two spaces)

Author: witgo <[email protected]>

Closes apache#480 from witgo/format_pom and squashes the following commits:

03f652f [witgo] review commit
b452680 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
bee920d [witgo] revert fix SPARK-1629: Spark Core missing commons-lang dependence
7382a07 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
6902c91 [witgo] fix SPARK-1629: Spark Core missing commons-lang dependence
0da4bc3 [witgo] merge master
d1718ed [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
e345919 [witgo] add avro dependency to yarn-alpha
77fad08 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
62d0862 [witgo] Fix org.scala-lang: * inconsistent versions dependency
1a162d7 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom
934f24d [witgo] review commit
cf46edc [witgo] exclude jruby
06e7328 [witgo] Merge branch 'SparkBuild' into format_pom
99464d2 [witgo] fix maven hadoop-provided profile fails to build
0c6c1fc [witgo] Fix compile spark core error with hadoop 0.23.x
6851bec [witgo] Maintain consistent SparkBuild.scala, pom.xml
andrewor14 pushed a commit to andrewor14/spark that referenced this pull request Jan 8, 2015
Handful of 0.9 fixes

This patch addresses a few fixes for Spark 0.9.0 based on the last release candidate.

@mridulm gets credit for reporting most of the issues here. Many of the fixes here are based on his work in apache#477 and follow up discussion with him.
(cherry picked from commit 77b986f)

Signed-off-by: Patrick Wendell <[email protected]>
yifeih pushed a commit to yifeih/spark that referenced this pull request Feb 20, 2019
###### _excavator_ is a bot for automating changes across repositories.

Changes produced by the roomba/latest-gradle-wrapper-oss check.

{runtimeCheckDesc}
To enable or disable this check, please contact the maintainers of Excavator.
bzhaoopenstack pushed a commit to bzhaoopenstack/spark that referenced this pull request Sep 11, 2019
This patch fixes the issue mentioned in PR#1519.

The default size of devstack swift is 2G, if we enable octavia or we
create some new images into glance during test. We will get failure.
The key reason for this is the swift device size is too small. So this
patch will increase the size through pass a env parameter in devstack
local conf.
Close: theopenlab/openlab#222
arjunshroff pushed a commit to arjunshroff/spark that referenced this pull request Nov 24, 2020
RolatZhang pushed a commit to RolatZhang/spark that referenced this pull request Aug 15, 2022
RolatZhang pushed a commit to RolatZhang/spark that referenced this pull request Aug 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants