[SPARK-44113][BUILD][INFRA][DOCS] Drop support for Scala 2.12 #43008

LuciferYang · 2023-09-20T06:27:47Z

What changes were proposed in this pull request?

The main purpose of this PR is to remove support for Scala 2.12 in Apache Spark 4.0, the specific work includes:

Run dev/change-scala-version.sh 2.13 to change the Scala version in all pom.xml files to 2.13
Cleaned up the parent pom.xml configuration related to Scala 2.12 and set the Scala 2.13 configuration as the default
Cleaned up the Scala 2.12 configuration in SparkBuild.scala
Cleaned up the support for Scala 2.12 in python/run-tests.py and dev/run-tests.py
Updated the sections related to Scala 2.12 in docs/building-spark.md, docs/index.md,docs/spark-connect-overview.md,docs/storage-openstack-swift.md,docs/_config.yml to Scala 2.13
Updated the parts related to Scala 2.12 in dev/test-dependencies.sh, dev/scalafmt, dev/mima,dev/lint-scala to Scala 2.13
Removed the support for Scala 2.12 in dev/change-scala-version.sh and cleaned up its invocation in Spark code
Updated dev/deps/spark-deps-hadoop-3-hive-2.3 and LICENSE-binary
Removed task scala-213 from build_and_test.yml because the daily test of other branches will not run this task, and the master branch already uses Scala 2.13 by default.
Replaced Scala 2.12 with Scala 2.13 in the name of the following .yml files: build_ansi.yml, build_coverage.yml, build_java11.yml,build_java17.yml, build_java21.yml,build_maven.yml, and build_rockdb_as_ui_backend.yml
Removed the support for Scala 2.12 in benchmark.yml
Moved files from src/scala-2.13/ directory to src/main/scala and deleted all files in the src/scala-2.12/ directory.
Comment out the code related to multiple Scala versions in load-spark-env.cmd and load-spark-env.sh.
Clean up redundant build-helper-maven-plugin configurations from core/pom.xml, repl/pom.xml, sql/api/pom.xml, sql/catalyst/pom.xml, sql/core/pom.xml and connector/connect/common/pom.xml

Why are the changes needed?

The minimum supported Scala version for Apache Spark 4.0 is Scala 2.13.

Does this PR introduce any user-facing change?

Yes, Apache will no longer support Scala 2.12

How was this patch tested?

Pass GitHub Actions

Was this patch authored or co-authored using generative AI tooling?

No

LuciferYang · 2023-09-20T08:17:14Z

dev/change-scala-version.sh

@@ -19,7 +19,7 @@

 set -e

-VALID_VERSIONS=( 2.12 2.13 )
+VALID_VERSIONS=( 2.13 )


No further modifications were made to this file, just made Scala-2.12 an invalid option. We can refactor this script in the future when we need to support multiple Scala versions again.

LuciferYang · 2023-09-20T08:19:55Z

cc @dongjoon-hyun FYI

Do we need to further split this PR ?

If you think this PR is too bloated, I can split
12. Moved files from src/scala-2.13/ directory to src/main/scala and deleted all files in the src/scala-2.12/ directory.
13. Comment out the code related to multiple Scala versions in load-spark-env.cmd and load-spark-env.sh.
14. Clean up redundant build-helper-maven-plugin configurations from core/pom.xml, repl/pom.xml, sql/api/pom.xml, sql/catalyst/pom.xml, sql/core/pom.xml and connector/connect/common/pom.xml

into separate prs.

dongjoon-hyun · 2023-09-20T19:48:31Z

I'm good with this single PR. Could you resolve the conflict?

LuciferYang · 2023-09-21T12:47:50Z

I'm good with this single PR. Could you resolve the conflict?

done ~

dongjoon-hyun · 2023-09-21T15:52:53Z

cc @HyukjinKwon , @srowen

dongjoon-hyun

+1, LGTM (from my side)

srowen

Yeah I think this is OK if tests pass. We can next look at making minor code changes that were hard or not possible while supporting 2.13, on the way to looking at supporting Scala 3.

dongjoon-hyun · 2023-09-21T21:58:47Z

Could you re-trigger the failed pipeline, @LuciferYang ?

LuciferYang · 2023-09-22T03:07:07Z

Could you re-trigger the failed pipeline, @LuciferYang ?

GA passed： https://github.com/apache/spark/runs/17002787095

pan3793 · 2023-09-22T03:28:16Z

dev/deps/spark-deps-hadoop-3-hive-2.3

+jline/3.22.0//jline-3.22.0.jar
+jna/5.13.0//jna-5.13.0.jar


why do they become runtime deps now?

And since both jline 2 and 3 are present at the classpath, is it safe?

Yes, I saw this dependency before, It is a cascading dependency of Scala 2.13. I think it is safe because the package name is different

jline2 is jline

jline3 is org.jline

it's fine then, thanks for checking

docs/building-spark.md

Co-authored-by: Cheng Pan <[email protected]>

dongjoon-hyun · 2023-09-22T04:03:42Z

Merged to master. Thank you, @LuciferYang and all.

HyukjinKwon · 2023-09-22T04:37:28Z

docs/_plugins/copy_api_dirs.rb

@@ -26,8 +26,8 @@
    curr_dir = pwd
    cd("..")

-    puts "Running 'build/sbt -Pkinesis-asl clean compile unidoc' from " + pwd + "; this may take a few minutes..."
-    system("build/sbt -Pkinesis-asl clean compile unidoc") || raise("Unidoc generation failed")
+    puts "Running 'build/sbt -Pscala-2.13 -Pkinesis-asl clean compile unidoc' from " + pwd + "; this may take a few minutes..."


QQ, why do we need scala profile here?

When we execute dev/change-scala-version.sh 2.13, it simultaneously modifies both copy_api_dirs.rb and mima files, adding these -Pscala-2.13. In this PR, I didn't clean them up. It seems we could add a follow-up to clear these unnecessary -Pscala-2.13.

### What changes were proposed in this pull request? This PR is a followup of #43008 that removes the leftover scheduled GitHub Actions build for Scala 2.13 scheduled build. ### Why are the changes needed? After dropping Scala 2.12, the default build is exactly same as the scheduled job for Scala 2.13 now. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Will monitor the scheduled builds. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #43047 from HyukjinKwon/SPARK-44113-folliwup. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

LuciferYang · 2023-09-22T04:59:41Z

Thanks all ~

…nts, and scala-2.13 profile usage ### What changes were proposed in this pull request? This PR is a followup of #43008 that cleanups Scala version specific comments, and `scala-2.13` profile usage. See also #43008 (comment) ### Why are the changes needed? To remove unnecessary profile specifications ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI in this PR should test them out. For README.md, I manually checked via GitHub. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #43049 from HyukjinKwon/SPARK-44113-followup-2. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

LuciferYang added 5 commits September 19, 2023 19:44

for maven

9fb0fb0

init

9e40ff2

init

43b45e9

remove

1b36dd4

fix md and yml

05e2db7

LuciferYang marked this pull request as draft September 20, 2023 06:27

github-actions bot added SQL MLLIB STRUCTURED STREAMING KUBERNETES GRAPHX MESOS BUILD SPARK SHELL YARN EXAMPLES DOCS CORE INFRA PYTHON DSTREAM AVRO CONNECT PROTOBUF labels Sep 20, 2023

LuciferYang changed the title ~~[SPARK-44113][BUILD] Drop Scala 2.12 Support~~ [WIP][SPARK-44113][BUILD] Drop Scala 2.12 Support Sep 20, 2023

LuciferYang changed the title ~~[WIP][SPARK-44113][BUILD] Drop Scala 2.12 Support~~ [WIP][SPARK-44113][BUILD] Drop support for Scala 2.12 Sep 20, 2023

LuciferYang changed the title ~~[WIP][SPARK-44113][BUILD] Drop support for Scala 2.12~~ [WIP][SPARK-44113][BUILD][INFRA][DOCS] Drop support for Scala 2.12 Sep 20, 2023

move Scala 2.13 code and remove scala-2.12 dir

21cab92

LuciferYang mentioned this pull request Sep 20, 2023

[SPARK-44112][BUILD][INFRA][DOCS] Drop support for Java 8 and Java 11 #43005

Closed

LuciferYang commented Sep 20, 2023

View reviewed changes

LuciferYang changed the title ~~[WIP][SPARK-44113][BUILD][INFRA][DOCS] Drop support for Scala 2.12~~ [SPARK-44113][BUILD][INFRA][DOCS] Drop support for Scala 2.12 Sep 20, 2023

LuciferYang marked this pull request as ready for review September 20, 2023 08:19

fix load-spark-env.cmd and load-spark-env.sh

6b5379c

github-actions bot added the WINDOWS label Sep 20, 2023

Clean up redundant build-helper-maven-plugin configurations.

285453f

Merge branch 'upmaster' into SPARK-44113

e256ab5

dongjoon-hyun approved these changes Sep 21, 2023

View reviewed changes

srowen approved these changes Sep 21, 2023

View reviewed changes

pan3793 reviewed Sep 22, 2023

View reviewed changes

docs/building-spark.md Outdated Show resolved Hide resolved

Update docs/building-spark.md

f5b1823

Co-authored-by: Cheng Pan <[email protected]>

pan3793 approved these changes Sep 22, 2023

View reviewed changes

dongjoon-hyun closed this in 3429202 Sep 22, 2023

HyukjinKwon reviewed Sep 22, 2023

View reviewed changes

HyukjinKwon mentioned this pull request Sep 22, 2023

[SPARK-44113][INFRA][FOLLOW-UP] Remove Scala 2.13 scheduled jobs #43047

Closed

HyukjinKwon mentioned this pull request Sep 22, 2023

[SPARK-45272][BUILD][INFRA][DOCS] Remove Scala version specific comments, and scala-2.13 profile usage #43049

Closed

yikf mentioned this pull request Sep 26, 2023

Use scala.jdk.CollectionConverters instead of scala.collection.JavaCo… yikf/spark#7

Closed

pan3793 mentioned this pull request Oct 23, 2023

[AUTHZ] Authz shaded include the transitive jackson libs apache/kyuubi#5496

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-44113][BUILD][INFRA][DOCS] Drop support for Scala 2.12 #43008

[SPARK-44113][BUILD][INFRA][DOCS] Drop support for Scala 2.12 #43008

LuciferYang commented Sep 20, 2023 •

edited

Loading

LuciferYang Sep 20, 2023

LuciferYang commented Sep 20, 2023 •

edited

Loading

dongjoon-hyun commented Sep 20, 2023

LuciferYang commented Sep 21, 2023

dongjoon-hyun commented Sep 21, 2023

dongjoon-hyun left a comment

srowen left a comment

dongjoon-hyun commented Sep 21, 2023

LuciferYang commented Sep 22, 2023 •

edited

Loading

pan3793 Sep 22, 2023

LuciferYang Sep 22, 2023

pan3793 Sep 22, 2023

dongjoon-hyun commented Sep 22, 2023

HyukjinKwon Sep 22, 2023

LuciferYang Sep 22, 2023

LuciferYang commented Sep 22, 2023

		jline/3.22.0//jline-3.22.0.jar
		jna/5.13.0//jna-5.13.0.jar

[SPARK-44113][BUILD][INFRA][DOCS] Drop support for Scala 2.12 #43008

[SPARK-44113][BUILD][INFRA][DOCS] Drop support for Scala 2.12 #43008

Conversation

LuciferYang commented Sep 20, 2023 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

LuciferYang Sep 20, 2023

Choose a reason for hiding this comment

LuciferYang commented Sep 20, 2023 • edited Loading

dongjoon-hyun commented Sep 20, 2023

LuciferYang commented Sep 21, 2023

dongjoon-hyun commented Sep 21, 2023

dongjoon-hyun left a comment

Choose a reason for hiding this comment

srowen left a comment

Choose a reason for hiding this comment

dongjoon-hyun commented Sep 21, 2023

LuciferYang commented Sep 22, 2023 • edited Loading

pan3793 Sep 22, 2023

Choose a reason for hiding this comment

LuciferYang Sep 22, 2023

Choose a reason for hiding this comment

pan3793 Sep 22, 2023

Choose a reason for hiding this comment

dongjoon-hyun commented Sep 22, 2023

HyukjinKwon Sep 22, 2023

Choose a reason for hiding this comment

LuciferYang Sep 22, 2023

Choose a reason for hiding this comment

LuciferYang commented Sep 22, 2023

LuciferYang commented Sep 20, 2023 •

edited

Loading

LuciferYang commented Sep 20, 2023 •

edited

Loading

LuciferYang commented Sep 22, 2023 •

edited

Loading