Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CELEBORN-1641] Pack the output spark client jar with the major.minor version #2881

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
+49 −43
Open

[CELEBORN-1641] Pack the output spark client jar with the major.minor version #2881

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bb7b5b0
add spark version
zaynt4606 Nov 5, 2024
16e9963
change version to major.minor version
zaynt4606 Nov 6, 2024
01e82b9
typo
zaynt4606 Nov 6, 2024
1f5ee9a
add spark2
zaynt4606 Nov 7, 2024
c3dbf24
add sbt spark version
zaynt4606 Nov 8, 2024
7e61e4e
md file change
zaynt4606 Nov 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions build/make-distribution.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,11 +185,10 @@ function build_spark_client {
| grep -v "WARNING" \
| tail -n 1)
SPARK_MAJOR_VERSION=${SPARK_VERSION%%.*}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unnecessary change

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

# Store the command as an array because $MVN variable might have spaces in it.
# Normal quoting tricks don't work.
# See: http://mywiki.wooledge.org/BashFAQ/050
BUILD_COMMAND=("$MVN" clean package $MVN_DIST_OPT -pl :celeborn-client-spark-${SPARK_MAJOR_VERSION}-shaded_$SCALA_VERSION -am $@)
BUILD_COMMAND=("$MVN" clean package $MVN_DIST_OPT -pl :celeborn-client-spark-${SPARK_VERSION}-shaded_$SCALA_VERSION -am $@)

# Actually build the jar
echo -e "\nBuilding with..."
Expand All @@ -200,7 +199,7 @@ function build_spark_client {
mkdir -p "$DIST_DIR/spark"

## Copy spark client jars
cp "$PROJECT_DIR"/client-spark/spark-$SPARK_MAJOR_VERSION-shaded/target/celeborn-client-spark-${SPARK_MAJOR_VERSION}-shaded_$SCALA_VERSION-$VERSION.jar "$DIST_DIR/spark/"
cp "$PROJECT_DIR"/client-spark/spark-${SPARK_MAJOR_VERSION}-shaded/target/celeborn-client-spark-${SPARK_VERSION}-shaded_$SCALA_VERSION-$VERSION.jar "$DIST_DIR/spark/"
}

function build_flink_client {
Expand Down
4 changes: 2 additions & 2 deletions client-spark/spark-2-shaded/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>celeborn-client-spark-2-shaded_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}-shaded_${scala.binary.version}</artifactId>
<packaging>jar</packaging>
<name>Celeborn Shaded Client for Spark 2</name>

<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-2_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand Down
2 changes: 1 addition & 1 deletion client-spark/spark-2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>celeborn-client-spark-2_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<packaging>jar</packaging>
<name>Celeborn Client for Spark 2</name>

Expand Down
4 changes: 2 additions & 2 deletions client-spark/spark-3-columnar-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>celeborn-spark-3-columnar-common_${scala.binary.version}</artifactId>
<artifactId>celeborn-spark-${spark.version}-columnar-common_${scala.binary.version}</artifactId>
<packaging>jar</packaging>
<name>Celeborn Client for Spark 3 Columnar Common</name>

<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
Expand Down
6 changes: 3 additions & 3 deletions client-spark/spark-3-columnar-shuffle/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>celeborn-spark-3-columnar-shuffle_${scala.binary.version}</artifactId>
<artifactId>celeborn-spark-${spark.version}-columnar-shuffle_${scala.binary.version}</artifactId>
<packaging>jar</packaging>
<name>Celeborn Client for Spark 3 Columnar Shuffle</name>

<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-spark-3-columnar-common_${scala.binary.version}</artifactId>
<artifactId>celeborn-spark-${spark.version}-columnar-common_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
Expand All @@ -49,7 +49,7 @@
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
Expand Down
14 changes: 7 additions & 7 deletions client-spark/spark-3-shaded/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>celeborn-client-spark-3-shaded_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}-shaded_${scala.binary.version}</artifactId>
<packaging>jar</packaging>
<name>Celeborn Shaded Client for Spark 3</name>

<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand Down Expand Up @@ -147,7 +147,7 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-spark-3-columnar-shuffle_${scala.binary.version}</artifactId>
<artifactId>celeborn-spark-${spark.version}-columnar-shuffle_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand All @@ -157,7 +157,7 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-spark-3-columnar-shuffle_${scala.binary.version}</artifactId>
<artifactId>celeborn-spark-${spark.version}-columnar-shuffle_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand All @@ -167,7 +167,7 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-spark-3-columnar-shuffle_${scala.binary.version}</artifactId>
<artifactId>celeborn-spark-${spark.version}-columnar-shuffle_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand All @@ -177,7 +177,7 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-spark-3-columnar-shuffle_${scala.binary.version}</artifactId>
<artifactId>celeborn-spark-${spark.version}-columnar-shuffle_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand All @@ -187,7 +187,7 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-spark-3-columnar-shuffle_${scala.binary.version}</artifactId>
<artifactId>celeborn-spark-${spark.version}-columnar-shuffle_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand Down
4 changes: 2 additions & 2 deletions client-spark/spark-3.5-columnar-shuffle/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-spark-3-columnar-common_${scala.binary.version}</artifactId>
<artifactId>celeborn-spark-${spark.version}-columnar-common_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
Expand All @@ -49,7 +49,7 @@
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
Expand Down
2 changes: 1 addition & 1 deletion client-spark/spark-3/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<packaging>jar</packaging>
<name>Celeborn Client for Spark 3</name>

Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ WorkerRef: null
Celeborn release binary contains clients for Spark 2.x and Spark 3.x, copy the corresponding client jar into Spark's
`jars/` directory:
```shell
cp $CELEBORN_HOME/spark/celeborn-client-spark-<spark.major.version>-shaded_<scala.binary.version>-<celeborn.version>.jar $SPARK_HOME/jars/
cp $CELEBORN_HOME/spark/celeborn-client-spark-<spark.version>-shaded_<scala.binary.version>-<celeborn.version>.jar $SPARK_HOME/jars/
```
#### Start spark-shell
Set `spark.shuffle.manager` to Celeborn's ShuffleManager, and turn off `spark.shuffle.service.enabled`:
Expand Down
2 changes: 1 addition & 1 deletion docs/deploy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ WorkerRef: null
Celeborn release binary contains clients for Spark 2.x and Spark 3.x, copy the corresponding client jar into Spark's
`jars/` directory:

Copy `$CELEBORN_HOME/spark/celeborn-client-spark-<spark.major.version>-shaded_<scala.binary.version>-<celeborn.version>.jar` to `$SPARK_HOME/jars/`.
Copy `$CELEBORN_HOME/spark/celeborn-client-spark-<spark.version>-shaded_<scala.binary.version>-<celeborn.version>.jar` to `$SPARK_HOME/jars/`.

### Spark Configuration
To use Celeborn, the following spark configurations should be added.
Expand Down
4 changes: 2 additions & 2 deletions docs/developers/glutensupport.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ mvn clean package -Pbackends-velox -Pspark-3.3 -Pceleborn -DskipTests

Then add the Gluten and Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).

- Celeborn: `celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar`
- Celeborn: `celeborn-client-spark-3.x.x-shaded_2.12-[celebornVersion].jar`
- Gluten: `gluten-velox-bundle-spark3.x_2.12-xx-xx-SNAPSHOT.jar` (The bundled Gluten Jar. Make sure -Pceleborn is specified when it is built.)

## ClickHouse Backend
Expand All @@ -48,7 +48,7 @@ mvn clean package -Pbackends-clickhouse -Pspark-3.3 -Pceleborn -DskipTests

Then add the Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).

- Celeborn: `celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar`
- Celeborn: `celeborn-client-spark-3.x.x-shaded_2.12-[celebornVersion].jar`

## Gluten Configuration

Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@
<maven.version>3.9.9</maven.version>

<flink.version>1.14.6</flink.version>
<spark.version>3.3.4</spark.version>
<spark.version>3.4.4</spark.version>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The IDEA Maven plugin prioritizes the outer properties in the artifactId.
Therefore, we need to update the version in the pom file when using the IDEA Maven plugin.
This update does not impact the compilation process.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spark version format should be majorVersion.minorVersion.bugfixVersion

How about using spark.binary.version property with majorVersion.minorVersion?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine with it~


<!-- use hadoop-3 as default -->
<hadoop.version>3.3.6</hadoop.version>
Expand Down
13 changes: 10 additions & 3 deletions project/CelebornBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -890,6 +890,13 @@ trait SparkClientProjects {
)
}

// the output would be something like: celeborn-client-spark-3.3.4-shaded_2.12-0.6.0-SNAPSHOT.jar
def sparkClientShadeJarName(
revision: String,
artifact: Artifact,
scalaBinaryVersionString: String): String =
s"celeborn-client-spark-$sparkVersion-shaded_$scalaBinaryVersionString" + "-" + revision + "." + artifact.extension

def sparkClientShade: Project = {
var p = Project(sparkClientShadedProjectName, file(sparkClientShadedProjectPath))
.dependsOn(sparkClient)
Expand All @@ -903,10 +910,10 @@ trait SparkClientProjects {
commonSettings,
releaseSettings,

// align final shaded jar name with maven.
(assembly / assemblyJarName) := {
val extension = artifact.value.extension
s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
val revision: String = version.value
val artifactValue: Artifact = artifact.value
sparkClientShadeJarName(revision, artifactValue, scalaBinaryVersion.value)
},

(assembly / test) := { },
Expand Down
28 changes: 14 additions & 14 deletions tests/spark-it/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,13 +97,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-2_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-2_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
Expand All @@ -115,13 +115,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
Expand All @@ -133,13 +133,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
Expand All @@ -151,13 +151,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
Expand All @@ -169,13 +169,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
Expand All @@ -187,13 +187,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
Expand All @@ -205,13 +205,13 @@
<dependencies>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.celeborn</groupId>
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
<artifactId>celeborn-client-spark-${spark.version}_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
Expand Down
Loading