SPARK-1119 and other build improvements

[pwendell]

1. Makes assembly and examples jar naming consistent in maven/sbt.
2. Updates make-distribution.sh to use Maven and fixes some bugs.
3. Updates the create-release script to call make-distribution script.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14365/

[witgo]

Why change using Maven build? The PR 480 has some relevant changes.

[pwendell]

@witgo we've been building releases with maven for several versions of spark due to better stability. We can look at #480 as well...

[pwendell]

Okay merged.

[witgo]

@berngp @pwendell ,

Whether we can delete the yarn.version, only using hadoop.version.
Like this,

 <profile>
      <id>yarn</id>
      <properties>
        <hadoop.major.version>2</hadoop.major.version>
        <hadoop.version>2.2.0</hadoop.version>
        <protobuf.version>2.5.0</protobuf.version>
      </properties>
      <modules>
        <module>yarn</module>
      </modules>
      <dependencyManagement>
        <dependencies>
          <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-api</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
              <exclusion>
                <groupId>asm</groupId>
                <artifactId>asm</artifactId>
              </exclusion>
              <exclusion>
                <groupId>org.ow2.asm</groupId>
                <artifactId>asm</artifactId>
              </exclusion>
              <exclusion>
                <groupId>org.jboss.netty</groupId>
                <artifactId>netty</artifactId>
              </exclusion>
            </exclusions>
          </dependency>
          <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-common</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
              <exclusion>
                <groupId>asm</groupId>
                <artifactId>asm</artifactId>
              </exclusion>
              <exclusion>
                <groupId>org.ow2.asm</groupId>
                <artifactId>asm</artifactId>
              </exclusion>
              <exclusion>
                <groupId>org.jboss.netty</groupId>
                <artifactId>netty</artifactId>
              </exclusion>
            </exclusions>
          </dependency>
          <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-client</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
              <exclusion>
                <groupId>asm</groupId>
                <artifactId>asm</artifactId>
              </exclusion>
              <exclusion>
                <groupId>org.ow2.asm</groupId>
                <artifactId>asm</artifactId>
              </exclusion>
              <exclusion>
                <groupId>org.jboss.netty</groupId>
                <artifactId>netty</artifactId>
              </exclusion>
            </exclusions>
          </dependency>
        </dependencies>
      </dependencyManagement>
    </profile>

will cause any problems?

[berngp]

Yes it will, not every one uses the same version of HDFS vs YARN.

On Wednesday, April 23, 2014, Li Guoqiang [email protected] wrote:

@berngp https://github.com/berngp @pwendellhttps://github.com/pwendell,
Whether we can delete the yarn.version, only using hadoop.version. This
will cause any problems?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/502#issuecomment-41237973
.

[witgo]

@berngp
Most of the people uses the same version of HDFS vs YARN.
We can be so

   <hadoop.version>1.0.4</hadoop.version>
   <yarn.version>${hadoop.version}</yarn.version>

 <profile>
      <id>yarn-alpha</id>
      <properties>
        <hadoop.major.version>2</hadoop.major.version>   
        <hadoop.version>0.23.7</hadoop.version>
      </properties>
      <modules>
        <module>yarn</module>
      </modules>
      <dependencyManagement>
        <dependencies>
          <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-api</artifactId>
            <version>${yarn.version}</version>
          </dependency>
          <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-common</artifactId>
            <version>${yarn.version}</version>
          </dependency>
          <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-client</artifactId>
            <version>${yarn.version}</version>
          </dependency>
        </dependencies>
      </dependencyManagement>
    </profile>

Most of the people use mvn -Pyarn -Dhadoop.version=2.3.0 -DskipTests clean package.
Others usemvn -Pyarn-alpha -Dhadoop.version=2.3.0 -DskipTests -Dyarn.version=0.23.9 clean package.

[berngp]

Makes sense to me.

On Wednesday, April 23, 2014, Li Guoqiang [email protected] wrote:

@berngp https://github.com/berngp
Most of the people uses the same version of HDFS vs YARN.
We can be so

<hadoop.version>1.0.4</hadoop.version>
<yarn.version>${hadoop.version}</yarn.version>

yarn-alpha 2 0.23.7 yarn org.apache.hadoop hadoop-yarn-api ${yarn.version} org.apache.hadoop hadoop-yarn-common ${yarn.version} org.apache.hadoop hadoop-yarn-client ${yarn.version}

Most of the people use mvn -Pyarn -Dhadoop.version=2.3.0 -DskipTests
clean package.
Others usemvn -Pyarn -Dhadoop.version=2.3.0 -DskipTests
-Dyarn.version=0.23.9 clean package.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/502#issuecomment-41239453
.