[SPARK-44112][BUILD][INFRA][DOCS] Drop support for Java 8 and Java 11 #43005

LuciferYang · 2023-09-20T03:29:14Z

What changes were proposed in this pull request?

The main purpose of this pr is to remove support for Java 8 and Java 11 in Apache Spark 4.0, the specific work includes:

pom.xml: change java.version from 1.8 to 17, change target:jvm-1.8 to target:17
SparkBuild.scala: change -target:jvm-${javaVersion.value} to -target:${javaVersion.value}
workflow files: change the default Java version from 8 to 17, and ensure that branch-3.x still uses Java 8. Removed the daily job for Java 11 and Java 17.
docs: replace parts of Java 8 and 11 with Java 17.
infra/Dockerfile: change to install openjdk-17-jdk-headless

Why are the changes needed?

The minimum supported Java version for Apache Spark 4.0 is Java 17

Does this PR introduce any user-facing change?

Yes, Apache will no longer support Java 8 and Java 11

How was this patch tested?

Pass Github Actions

Was this patch authored or co-authored using generative AI tooling?

No

dongjoon-hyun

Thank you, @LuciferYang .

This reverts commit cb5f8de.

This reverts commit 72b51ad.

LuciferYang · 2023-09-20T05:42:36Z

@dongjoon-hyun I'd like to discuss with you that https://github.com/LuciferYang/spark/actions/runs/6243680485/job/16949769483 has tested Java 17 + Scala 2.12.18, and a lot of tests failed after changing -target:jvm-1.8 to -target:17.

I can reproduce these errors locally,

java -version
openjdk version "17.0.8" 2023-07-18 LTS
OpenJDK Runtime Environment Zulu17.44+15-CA (build 17.0.8+7-LTS)
OpenJDK 64-Bit Server VM Zulu17.44+15-CA (build 17.0.8+7-LTS, mixed mode, sharing)

build/sbt clean "catalyst/test" -Pscala-2.12

[info] LimitPushdownSuite:
[info] org.apache.spark.sql.catalyst.optimizer.LimitPushdownSuite *** ABORTED *** (0 milliseconds)
[info]   java.lang.IllegalAccessError: Update to non-static final field org.apache.spark.SparkFunSuite.invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected attempted from a different method (org$scalatest$BeforeAndAfterAll$_setter_$invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected_$eq) than the initializer method <init>
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$_setter_$invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected_$eq(SparkFunSuite.scala:69)
[info]   at org.scalatest.BeforeAndAfterAll.$init$(BeforeAndAfterAll.scala:148)
[info]   at org.apache.spark.SparkFunSuite.<init>(SparkFunSuite.scala:70)
[info]   at org.apache.spark.sql.catalyst.optimizer.LimitPushdownSuite.<init>(LimitPushdownSuite.scala:29)
[info]   at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[info]   at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
[info]   at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[info]   at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
[info]   at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[info]   at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:347)
[info]   at java.base/java.lang.Class.newInstance(Class.java:645)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info]   at java.base/java.lang.Thread.run(Thread.java:833)
[error] Uncaught exception when running org.apache.spark.sql.catalyst.optimizer.LimitPushdownSuite: java.lang.IllegalAccessError: Update to non-static final field org.apache.spark.SparkFunSuite.invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected attempted from a different method (org$scalatest$BeforeAndAfterAll$_setter_$invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected_$eq) than the initializer method <init> 
[error] sbt.ForkMain$ForkError: java.lang.IllegalAccessError: Update to non-static final field org.apache.spark.SparkFunSuite.invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected attempted from a different method (org$scalatest$BeforeAndAfterAll$_setter_$invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected_$eq) than the initializer method <init> 
[error] 	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$_setter_$invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected_$eq(SparkFunSuite.scala:69)
[error] 	at org.scalatest.BeforeAndAfterAll.$init$(BeforeAndAfterAll.scala:148)
[error] 	at org.apache.spark.SparkFunSuite.<init>(SparkFunSuite.scala:70)
[error] 	at org.apache.spark.sql.catalyst.optimizer.LimitPushdownSuite.<init>(LimitPushdownSuite.scala:29)
[error] 	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error] 	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
[error] 	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error] 	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
[error] 	at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[error] 	at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:347)
[error] 	at java.base/java.lang.Class.newInstance(Class.java:645)
[error] 	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
[error] 	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] 	at java.base/java.lang.Thread.run(Thread.java:833)
...

[info] Run completed in 1 second, 569 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 297
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] *** 297 SUITES ABORTED ***
[error] Error: Total 331, Failed 19, Errors 297, Passed 15
[error] Failed tests:
[error] 	org.apache.spark.sql.catalyst.expressions.RowBasedKeyValueBatchSuite
[error] 	org.apache.spark.sql.connector.catalog.CatalogLoadingSuite
[error] Error during tests:
[error] 	org.apache.spark.sql.catalyst.optimizer.ReassignLambdaVariableIDSuite
[error] 	org.apache.spark.sql.catalyst.plans.logical.AnalysisHelperSuite
...

but when we change the Scala version to Scala 2.13 and then test with Java 17 again,

dev/change-scala-version.sh 2.13
build/sbt clean "catalyst/test" -Pscala-2.13

[info] Run completed in 2 minutes, 16 seconds.
[info] Total number of tests run: 7138
[info] Suites: completed 297, aborted 0
[info] Tests: succeeded 7138, failed 0, canceled 1, ignored 5, pending 0
[info] All tests passed.

all test successful.

If this is the case, can we drop Scala 2.12 supports first, then upgrade the Java version? This seems to be a bit easier.

dongjoon-hyun · 2023-09-20T05:44:51Z

+1 for switching to Scala 2.13 first. Please lead the activity. I trust your domain expertise. :)

If this is the case, can we drop Scala 2.12 supports first, then upgrade the Java version? This seems to be a bit easier.

LuciferYang · 2023-09-20T05:45:55Z

+1 for switching to Scala 2.13 first. Please lead the activity. I trust your domain expertise. :)

OK ~

LuciferYang · 2023-09-20T08:14:31Z

wait #43008

.github/workflows/build_coverage.yml

.github/workflows/publish_snapshot_java8.yml

dongjoon-hyun · 2023-09-22T04:07:29Z

#43008 is merged. Shall we resume this?

LuciferYang · 2023-09-22T05:01:48Z

I'll update this PR later.

LuciferYang · 2023-09-22T05:13:09Z

docs/building-spark.md

@@ -27,7 +27,7 @@ license: |
 ## Apache Maven

 The Maven-based build is the build of reference for Apache Spark.
-Building Spark using Maven requires Maven 3.9.4 and Java 8/11/17.
+Building Spark using Maven requires Maven 3.9.4 and Java 17.


Should we add Support for Java 8/11 was removed in Spark 4.0.0.

LuciferYang · 2023-09-22T05:15:50Z

.github/workflows/build_and_test.yml

@@ -649,11 +649,11 @@ jobs:
        if [ -f ./dev/free_disk_space_container ]; then
          ./dev/free_disk_space_container
        fi
-    - name: Install Java 8
+    - name: Install Java 17


I need to think about this part a bit more. I just recalled that branch-3.x will also use this yml, so we may need to add some if conditions.

LuciferYang · 2023-09-22T05:37:50Z

.github/workflows/build_and_test.yml

@@ -649,11 +649,11 @@ jobs:
        if [ -f ./dev/free_disk_space_container ]; then
          ./dev/free_disk_space_container
        fi
-    - name: Install Java 8
+    - name: Install Java ${{ inputs.java }}


It seems easier to install Java this way, the daily test of branch-3.x all with java: 8

LuciferYang · 2023-09-22T08:30:49Z

Work not included in this PR:

Dockerfile: including dev/create-release/spark-rm/Dockerfile and connector/docker/spark-test/base/Dockerfile
appveyor-install-dependencies.ps1: Need to install a Java 17 for windows SparkR test(do independent testing on Test Appveyor use pre-installed java 17 #43054)

Since I can't successfully run docker build -t spark-rm --build-arg UID=$UID on my M2 Max now, I'm posting to verify if my modifications are correct. On the other hand, I also don't know how to install Java 17 in appveyor, so I need to do further investigation.

So, I am planning to create two additional Jira tickets to track these two issues, is that okay?

LuciferYang · 2023-09-22T08:45:25Z

Could you review this @dongjoon-hyun @HyukjinKwon @srowen Thanks

LuciferYang · 2023-09-22T09:13:57Z

dev/infra/Dockerfile

@@ -30,7 +30,7 @@ RUN apt-get update && apt-get install -y \
    pkg-config  \
    curl  \
    wget  \
-    openjdk-8-jdk  \
+    openjdk-17-jdk-headless  \


@dongjoon-hyun @HyukjinKwon should we change this?

@dongjoon-hyun So in this PR, should we revert this change and consider it as a followup?

Emm, jdk install in here seems useless, because we are using github action to install the java. But if CI passed, it is ok.

Thanks @Yikun ~ Let me continue to monitor the running status of GA.

dongjoon-hyun

+1, the AS-IS PR looks good to me, @LuciferYang . We can do the follow-ups.

The Spark release docker files
Infra docker files
AppVeyor
Other documentations including README.md

dongjoon-hyun · 2023-09-22T12:35:40Z

IIUC, the main change passed in c0a330d already.

dongjoon-hyun · 2023-09-22T12:37:33Z

Merged to master for Apache Spark 4.0.0. Thank you, @LuciferYang and all.

LuciferYang · 2023-09-22T12:54:17Z

Thanks @dongjoon-hyun @HyukjinKwon @bjornjorgensen and @cfmcgrady ~

### What changes were proposed in this pull request? This pr aims change Maven daily test use Java 17 for testing. ### Why are the changes needed? #43005 drop support for Java 8 and Java 11, so Maven daily test should also test with Java 17 as default. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Monitor Maven daily test GA task ### Was this patch authored or co-authored using generative AI tooling? No Closes #43057 from LuciferYang/SPARK-45280. Authored-by: yangjie01 <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? This PR aims to fix `docker-image-tool.sh` to be up-to-date. ### Why are the changes needed? Apache Spark 4 dropped Java 11 support. So, we should fix the following. - #43005 ``` - - Build and push Java11-based image with tag "v3.4.0" to docker.io/myrepo + - Build and push Java17-based image with tag "v4.0.0" to docker.io/myrepo ``` Apache Spark 4 requires JDK instead of JRE. So, we should fix the following. - #45761 ``` - $0 -r docker.io/myrepo -t v3.4.0 -b java_image_tag=11-jre build + $0 -r docker.io/myrepo -t v4.0.0 -b java_image_tag=17 build ``` Lastly, `3.4.0` is too old because it's released on April 13, 2023. We had better use v4.0.0. ``` - $0 -r docker.io/myrepo -t v3.4.0 -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile build + $0 -r docker.io/myrepo -t v4.0.0 -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile build ``` ### Does this PR introduce _any_ user-facing change? No functional change because this is a usage message. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #47618 from dongjoon-hyun/SPARK-49117. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

### What changes were proposed in this pull request? This PR aims to fix `docker-image-tool.sh` to be up-to-date. ### Why are the changes needed? Apache Spark 4 dropped Java 11 support. So, we should fix the following. - apache#43005 ``` - - Build and push Java11-based image with tag "v3.4.0" to docker.io/myrepo + - Build and push Java17-based image with tag "v4.0.0" to docker.io/myrepo ``` Apache Spark 4 requires JDK instead of JRE. So, we should fix the following. - apache#45761 ``` - $0 -r docker.io/myrepo -t v3.4.0 -b java_image_tag=11-jre build + $0 -r docker.io/myrepo -t v4.0.0 -b java_image_tag=17 build ``` Lastly, `3.4.0` is too old because it's released on April 13, 2023. We had better use v4.0.0. ``` - $0 -r docker.io/myrepo -t v3.4.0 -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile build + $0 -r docker.io/myrepo -t v4.0.0 -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile build ``` ### Does this PR introduce _any_ user-facing change? No functional change because this is a usage message. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47618 from dongjoon-hyun/SPARK-49117. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

The main purpose of this pr is to remove support for Java 8 and Java 11 in Apache Spark 4.0, the specific work includes: 1. `pom.xml`: change `java.version` from 1.8 to 17, change `target:jvm-1.8` to `target:17` 2. `SparkBuild.scala`: change `-target:jvm-${javaVersion.value}` to `-target:${javaVersion.value}` 3. workflow files: change the default Java version from 8 to 17, and ensure that branch-3.x still uses Java 8. Removed the daily job for Java 11 and Java 17. 4. docs: replace parts of Java 8 and 11 with Java 17. The minimum supported Java version for Apache Spark 4.0 is Java 17 Yes, Apache will no longer support Java 8 and Java 11 - Pass Github Actions No Closes apache#43005 from LuciferYang/SPARK-44112. Authored-by: yangjie01 <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

LuciferYang added 2 commits September 20, 2023 11:27

init

4079d7a

add old branch publish_snapshot

4c11396

LuciferYang marked this pull request as draft September 20, 2023 03:29

github-actions bot added BUILD DOCS INFRA labels Sep 20, 2023

LuciferYang changed the title ~~[SPARK-44112][BUILD][INFRA] Drop Java 8 and 11 support~~ [WIP][SPARK-44112][BUILD][INFRA] Drop Java 8 and 11 support Sep 20, 2023

remove 11 maven build

f7ae713

dongjoon-hyun reviewed Sep 20, 2023

View reviewed changes

LuciferYang added 4 commits September 20, 2023 13:20

test with Scala 2.13

cb5f8de

Revert "test with Scala 2.13"

72b51ad

This reverts commit cb5f8de.

SparkBuild

4c6f254

Revert "Revert "test with Scala 2.13""

50d7693

This reverts commit 72b51ad.

tpcds test scala 2.13

7ce64d8

bjornjorgensen reviewed Sep 20, 2023

View reviewed changes

.github/workflows/build_coverage.yml Outdated Show resolved Hide resolved

fix comments

bbeba9a

cfmcgrady reviewed Sep 21, 2023

View reviewed changes

.github/workflows/publish_snapshot_java8.yml Outdated Show resolved Hide resolved

LuciferYang added 2 commits September 21, 2023 20:43

merge publish snapshot java 8 into publish snapshot

278cac6

Merge branch 'upmaster' into SPARK-44112

3f11512

Merge branch 'upmaster' into SPARK-44112

bbbed13

LuciferYang commented Sep 22, 2023

View reviewed changes

remove -Pscala-2.13

c0a330d

LuciferYang commented Sep 22, 2023

View reviewed changes

change to use inputs.java

6ca32e3

LuciferYang commented Sep 22, 2023

View reviewed changes

add more doc

2ea3160

github-actions bot added the PYTHON label Sep 22, 2023

LuciferYang changed the title ~~[WIP][SPARK-44112][BUILD][INFRA] Drop Java 8 and 11 support~~ [WIP][SPARK-44112][BUILD][INFRA][DOCS] Drop Java 8 and 11 support Sep 22, 2023

LuciferYang changed the title ~~[WIP][SPARK-44112][BUILD][INFRA][DOCS] Drop Java 8 and 11 support~~ [WIP][SPARK-44112][BUILD][INFRA][DOCS] Drop support for Java 8 and Java 11 Sep 22, 2023

LuciferYang changed the title ~~[WIP][SPARK-44112][BUILD][INFRA][DOCS] Drop support for Java 8 and Java 11~~ [SPARK-44112][BUILD][INFRA][DOCS] Drop support for Java 8 and Java 11 Sep 22, 2023

infra Dockerfile install openjdk-17-jdk-headless

54f40ab

LuciferYang commented Sep 22, 2023

View reviewed changes

LuciferYang marked this pull request as ready for review September 22, 2023 09:14

dongjoon-hyun approved these changes Sep 22, 2023

View reviewed changes

dongjoon-hyun closed this in 46ebed2 Sep 22, 2023

LuciferYang mentioned this pull request Sep 22, 2023

[SPARK-45280][INFRA] Change Maven daily test use Java 17 for testing #43057

Closed

harupy mentioned this pull request Sep 25, 2023

Fix spark dev java version mlflow/mlflow#9707

Merged

37 tasks

danepitkin mentioned this pull request Oct 5, 2023

[Java] Remove Java 8 support in Arrow v18 apache/arrow#38051

Closed

7 tasks

dongjoon-hyun mentioned this pull request Aug 5, 2024

[SPARK-49117][K8S] Fix docker-image-tool.sh to be up-to-date #47618

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-44112][BUILD][INFRA][DOCS] Drop support for Java 8 and Java 11 #43005

[SPARK-44112][BUILD][INFRA][DOCS] Drop support for Java 8 and Java 11 #43005

LuciferYang commented Sep 20, 2023 •

edited

Loading

dongjoon-hyun left a comment

LuciferYang commented Sep 20, 2023

dongjoon-hyun commented Sep 20, 2023

LuciferYang commented Sep 20, 2023

LuciferYang commented Sep 20, 2023

dongjoon-hyun commented Sep 22, 2023

LuciferYang commented Sep 22, 2023

LuciferYang Sep 22, 2023

LuciferYang Sep 22, 2023

LuciferYang Sep 22, 2023

LuciferYang commented Sep 22, 2023 •

edited

Loading

LuciferYang commented Sep 22, 2023

LuciferYang Sep 22, 2023

HyukjinKwon Sep 22, 2023

LuciferYang Sep 22, 2023

Yikun Sep 22, 2023

LuciferYang Sep 22, 2023

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun commented Sep 22, 2023

dongjoon-hyun commented Sep 22, 2023

LuciferYang commented Sep 22, 2023

[SPARK-44112][BUILD][INFRA][DOCS] Drop support for Java 8 and Java 11 #43005

[SPARK-44112][BUILD][INFRA][DOCS] Drop support for Java 8 and Java 11 #43005

Conversation

LuciferYang commented Sep 20, 2023 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

dongjoon-hyun left a comment

Choose a reason for hiding this comment

LuciferYang commented Sep 20, 2023

dongjoon-hyun commented Sep 20, 2023

LuciferYang commented Sep 20, 2023

LuciferYang commented Sep 20, 2023

dongjoon-hyun commented Sep 22, 2023

LuciferYang commented Sep 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LuciferYang commented Sep 22, 2023 • edited Loading

LuciferYang commented Sep 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

dongjoon-hyun commented Sep 22, 2023

dongjoon-hyun commented Sep 22, 2023

LuciferYang commented Sep 22, 2023

LuciferYang commented Sep 20, 2023 •

edited

Loading

LuciferYang commented Sep 22, 2023 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading