Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add spark2.4.6 to maven-publish-scala #5138

Merged
merged 29 commits into from
Jul 26, 2022
Merged

add spark2.4.6 to maven-publish-scala #5138

Changes from 10 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
3ae03e2
add spark2.4.6 build with maven
pinggao18 Jul 20, 2022
253b7a1
update
pinggao18 Jul 20, 2022
f6b2c65
update
pinggao18 Jul 20, 2022
f5edd53
update
pinggao18 Jul 20, 2022
ff8671d
update
pinggao18 Jul 20, 2022
5eba2b3
update
pinggao18 Jul 20, 2022
473611c
update
pinggao18 Jul 20, 2022
fa32a2a
update
pinggao18 Jul 21, 2022
e4cd8bf
update
pinggao18 Jul 21, 2022
4615186
update
pinggao18 Jul 21, 2022
3bf8fd6
update
pinggao18 Jul 21, 2022
6f19485
update
pinggao18 Jul 21, 2022
8d99d8d
update
pinggao18 Jul 21, 2022
6501ab6
update
pinggao18 Jul 21, 2022
77cad0a
update
pinggao18 Jul 21, 2022
391cbf3
update
pinggao18 Jul 21, 2022
5d2c31e
update
pinggao18 Jul 21, 2022
26311b9
update
pinggao18 Jul 21, 2022
9579df3
update
pinggao18 Jul 21, 2022
9ad1ef9
update
pinggao18 Jul 21, 2022
9b3e8ce
update
pinggao18 Jul 21, 2022
5324acf
update
pinggao18 Jul 21, 2022
414cf4e
update
pinggao18 Jul 21, 2022
12264cf
update
pinggao18 Jul 21, 2022
3b2b3cf
update
pinggao18 Jul 26, 2022
ac054a4
update
pinggao18 Jul 26, 2022
749ace2
update
pinggao18 Jul 26, 2022
16ad964
update
pinggao18 Jul 26, 2022
53554ca
update
pinggao18 Jul 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 66 additions & 2 deletions .github/workflows/maven-publish-scala.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
name: Maven Nightly Build Publish Scala

on:
pull_request:
branches: [ "main" ]
#release:
# types: [created]
schedule:
Expand Down Expand Up @@ -42,6 +44,9 @@ jobs:

- name: Set up Maven Settings
uses: s4u/[email protected]
env:
OSSRH_USERNAME: ${{ secrets.OSSRH_USERNAME }}
OSSRH_PASSWORD: ${{ secrets.OSSRH_PASSWORD }}
with:
sonatypeSnapshots: true
apacheSnapshots: true
Expand All @@ -58,8 +63,8 @@ jobs:
}
},{
"id": "ossrh",
"username": "${{ secrets.OSSRH_USERNAME }}",
"password": "${{ secrets.OSSRH_PASSWORD }}",
"username": "${OSSRH_USERNAME}",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put this in
env:

like

  • name: Do something
    env:
    FOO_TOKEN: ${{ secrets.FOO_TOKEN }}
    run: |
    echo "FOO_TOKEN = ${FOO_TOKEN}"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put this in env:

like

  • name: Do something
    env:
    FOO_TOKEN: ${{ secrets.FOO_TOKEN }}
    run: |
    echo "FOO_TOKEN = ${FOO_TOKEN}"

test:

- name: Set up OSSRH
      env:
        OSSRH_USERNAME: ${{ secrets.OSSRH_USERNAME }}
        OSSRH_PASSWORD: ${{ secrets.OSSRH_PASSWORD }}
      run: |
        echo "OSSRH_USERNAME = ${OSSRH_USERNAME}"
        echo "OSSRH_PASSWORD = ${OSSRH_PASSWORD}"

still can't get it

Run echo "OSSRH_USERNAME = ${OSSRH_USERNAME}"
OSSRH_USERNAME = 
OSSRH_PASSWORD =

"password": "${{OSSRH_PASSWORD}",
"configuration": {
"timeout": "3600000"
}
Expand All @@ -70,6 +75,18 @@ jobs:
run: |
ls
cat ~/.m2/settings.xml
#spark3.1.2
cp scala/pom.xml scala/pom.xml.origin
cp scala/common/spark-version/pom.xml scala/common/spark-version/pom.xml.origin
cp scala/common/spark-version/3.0/pom.xml scala/common/spark-version/3.0/pom.xml.origin
cp scala/dllib/pom.xml scala/dllib/pom.xml.origin
cp scala/orca/pom.xml scala/orca/pom.xml.origin
cp scala/friesian/pom.xml scala/friesian/pom.xml.origin
cp scala/grpc/pom.xml scala/grpc/pom.xml.origin
cp scala/serving/pom.xml scala/serving/pom.xml.origin
cp scala/ppml/pom.xml scala/ppml/pom.xml.origin
cp scala/assembly/pom.xml scala/assembly/pom.xml.origin

sed -i 's/<artifactId>${spark-version.project}<\/artifactId>/<artifactId>${spark-version.project}-${SPARK_PLATFORM}<\/artifactId>/' scala/dllib/pom.xml
sed -i 's/<artifactId>3.0<\/artifactId>/<artifactId>3.0-${SPARK_PLATFORM}<\/artifactId>/' scala/common/spark-version/3.0/pom.xml
sed -i 's/<artifactId>bigdl-parent-spark_${spark.version}<\/artifactId>/<artifactId>bigdl-parent-spark_3.1.2<\/artifactId>/' scala/pom.xml
Expand All @@ -83,6 +100,53 @@ jobs:
sed -i 's/<artifactId>bigdl-parent-spark_${spark.version}<\/artifactId>/<artifactId>bigdl-parent-spark_3.1.2<\/artifactId>/' scala/ppml/pom.xml
sed -i 's/<artifactId>bigdl-parent-spark_${spark.version}<\/artifactId>/<artifactId>bigdl-parent-spark_3.1.2<\/artifactId>/' scala/assembly/pom.xml
mvn -Dhttp.proxyHost=${{ secrets.HTTP_PROXY_HOST_2 }} -Dhttp.proxyPort=${{ secrets.HTTP_PROXY_PORT_2 }} -Dhttps.proxyHost=${{ secrets.HTTP_PROXY_HOST_2 }} -Dhttps.proxyPort=${{ secrets.HTTP_PROXY_PORT_3 }} clean deploy -DskipTests -Dspark.version=3.1.2 -DSPARK_PLATFORM=SPARK_3.1 -P spark_3.x --file scala/pom.xml

mv scala/pom.xml.origin scala/pom.xml
mv scala/common/spark-version/pom.xml.origin scala/common/spark-version/pom.xml
mv scala/common/spark-version/3.0/pom.xml.origin scala/common/spark-version/3.0/pom.xml
mv scala/dllib/pom.xml.origin scala/dllib/pom.xml
mv scala/orca/pom.xml.origin scala/orca/pom.xml
mv scala/friesian/pom.xml.origin scala/friesian/pom.xml
mv scala/grpc/pom.xml.origin scala/grpc/pom.xml
mv scala/serving/pom.xml.origin scala/serving/pom.xml
mv scala/ppml/pom.xml.origin scala/ppml/pom.xml
mv scala/assembly/pom.xml.origin scala/assembly/pom.xml
#spark2.4.6
cp scala/pom.xml scala/pom.xml.origin
cp scala/common/spark-version/pom.xml scala/common/spark-version/pom.xml.origin
cp scala/common/spark-version/2.0/pom.xml scala/common/spark-version/2.0/pom.xml.origin
cp scala/dllib/pom.xml scala/dllib/pom.xml.origin
cp scala/orca/pom.xml scala/orca/pom.xml.origin
cp scala/friesian/pom.xml scala/friesian/pom.xml.origin
cp scala/grpc/pom.xml scala/grpc/pom.xml.origin
cp scala/serving/pom.xml scala/serving/pom.xml.origin
cp scala/ppml/pom.xml scala/ppml/pom.xml.origin
cp scala/assembly/pom.xml scala/assembly/pom.xml.origin

sed -i 's/<artifactId>${spark-version.project}<\/artifactId>/<artifactId>${spark-version.project}-${SPARK_PLATFORM}<\/artifactId>/' scala/dllib/pom.xml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should cp some copies for the original pom since they need to be modified twice

sed -i 's/<artifactId>2.0<\/artifactId>/<artifactId>2.0-${SPARK_PLATFORM}<\/artifactId>/' scala/common/spark-version/2.0/pom.xml
sed -i 's/<artifactId>bigdl-parent-spark_${spark.version}<\/artifactId>/<artifactId>bigdl-parent-spark_2.4.6<\/artifactId>/' scala/pom.xml
sed -i 's/<artifactId>bigdl-parent-spark_${spark.version}<\/artifactId>/<artifactId>bigdl-parent-spark_2.4.6<\/artifactId>/' scala/common/spark-version/pom.xml
sed -i 's/<artifactId>bigdl-parent-spark_${spark.version}<\/artifactId>/<artifactId>bigdl-parent-spark_2.4.6<\/artifactId>/' scala/common/spark-version/2.0/pom.xml
sed -i 's/<artifactId>bigdl-parent-spark_${spark.version}<\/artifactId>/<artifactId>bigdl-parent-spark_2.4.6<\/artifactId>/' scala/dllib/pom.xml
sed -i 's/<artifactId>bigdl-parent-spark_${spark.version}<\/artifactId>/<artifactId>bigdl-parent-spark_2.4.6<\/artifactId>/' scala/orca/pom.xml
sed -i 's/<artifactId>bigdl-parent-spark_${spark.version}<\/artifactId>/<artifactId>bigdl-parent-spark_2.4.6<\/artifactId>/' scala/friesian/pom.xml
sed -i 's/<artifactId>bigdl-parent-spark_${spark.version}<\/artifactId>/<artifactId>bigdl-parent-spark_2.4.6<\/artifactId>/' scala/grpc/pom.xml
sed -i 's/<artifactId>bigdl-parent-spark_${spark.version}<\/artifactId>/<artifactId>bigdl-parent-spark_2.4.6<\/artifactId>/' scala/serving/pom.xml
sed -i 's/<artifactId>bigdl-parent-spark_${spark.version}<\/artifactId>/<artifactId>bigdl-parent-spark_2.4.6<\/artifactId>/' scala/ppml/pom.xml
sed -i 's/<artifactId>bigdl-parent-spark_${spark.version}<\/artifactId>/<artifactId>bigdl-parent-spark_2.4.6<\/artifactId>/' scala/assembly/pom.xml
mvn -Dhttp.proxyHost=${{ secrets.HTTP_PROXY_HOST_2 }} -Dhttp.proxyPort=${{ secrets.HTTP_PROXY_PORT_2 }} -Dhttps.proxyHost=${{ secrets.HTTP_PROXY_HOST_2 }} -Dhttps.proxyPort=${{ secrets.HTTP_PROXY_PORT_3 }} clean deploy -DskipTests -Dspark.version=2.4.6 -DSPARK_PLATFORM=SPARK_2.4 -P spark_2.x --file scala/pom.xml

mv scala/pom.xml.origin scala/pom.xml
mv scala/common/spark-version/pom.xml.origin scala/common/spark-version/pom.xml
mv scala/common/spark-version/2.0/pom.xml.origin scala/common/spark-version/2.0/pom.xml
mv scala/dllib/pom.xml.origin scala/dllib/pom.xml
mv scala/orca/pom.xml.origin scala/orca/pom.xml
mv scala/friesian/pom.xml.origin scala/friesian/pom.xml
mv scala/grpc/pom.xml.origin scala/grpc/pom.xml
mv scala/serving/pom.xml.origin scala/serving/pom.xml
mv scala/ppml/pom.xml.origin scala/ppml/pom.xml
mv scala/assembly/pom.xml.origin scala/assembly/pom.xml

# - name: Publish to GitHub Packages Apache Maven
# run: mvn deploy -s $GITHUB_WORKSPACE/settings.xml
Expand Down