Scala 2.13 Support #8592

NVnavkumar · 2023-06-21T04:41:28Z

This fixes the compilation errors, and provides the updates to the build system to enable spark-rapids to build with Scala 2.13.

You can build it by changing to the scala2.13 directory, and running maven:

cd scala2.13
mvn verify

You can also open the scala2.13 directory directly in IntelliJ IDE and it should work very similarly to 2.12.

See scala2.13/README.md for more information.

Signed-off-by: Navin Kumar [email protected]

Signed-off-by: Navin Kumar <[email protected]>

pom.xml

dev/change-scala-version.sh

pxLi

One question about the goal here, are we planning to do

multiple profiles to build different shims w/ different scalas, and we will deliver 2 sets of artifacts here (2.12 plugin for all spark shims, 2.13 plugin for 330+ shims)
Or build the plugin using scala 2.12 for spark shims 330-, and scala 2.13 for 330+ as default. We still just release one plugin artifact including 2.12 330- shims, 2.13 330+ shims, and 2.12/2.13 common parts

also cc @sameerz @GaryShen2008

sameerz · 2023-06-27T00:27:26Z

multiple profiles to build different shims w/ different scalas, and we will deliver 2 sets of artifacts here (2.12 plugin for all spark shims, 2.13 plugin for 330+ shims)

We will release 2 sets of artifacts, a scala 2.12 plugin for all spark shims, and a scala 2.13 plugin for 3.3.0+ shims.

pxLi · 2023-06-27T00:52:33Z

multiple profiles to build different shims w/ different scalas, and we will deliver 2 sets of artifacts here (2.12 plugin for all spark shims, 2.13 plugin for 330+ shims)

We will release 2 sets of artifacts, a scala 2.12 plugin for all spark shims, and a scala 2.13 plugin for 3.3.0+ shims.

Got it, thanks for the confirmation! I will follow up CI setup after this PR along w/ the internal private one

…n.Seq from Spark 3.4 and higher. This is to avoid unnecessary copying of data.

… of converting when returning Seq and using Seq in _* pattern

…a 2.13 and will cause compilation to fail Signed-off-by: Navin Kumar <[email protected]>

Remove checking scala2.13 folder, .github/workflows/mvn-verify-check.yml will check it Signed-off-by: Tim Liu <[email protected]>

NvTimLiu · 2023-10-26T04:00:06Z

@NVnavkumar Sorry I added dup git diff scala2.13 folder in the jenkins/Jenkinsfile-blossom.premerge

I noticed you've already add the git diff check for it in the .github/workflows/mvn-verify-check.yml file

So I pushed a commit here to revert it:
c88320b

NvTimLiu · 2023-10-26T05:28:59Z

build

NvTimLiu

LGTM

scala2.13/aggregator/pom.xml

scala2.13/README.md

tgravescs · 2023-10-26T14:57:31Z

scala2.13/README.md

+
+That way any new dependencies or other changes will be picked up in the Scala 2.13 build.
+
+## IDE Integration with IntelliJ


is this addressed did we file issue?

sql-plugin/src/main/scala-2.13/com/nvidia/spark/rapids/ArmScalaFixes.scala

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuAggregateExec.scala

sql-plugin/src/main/scala-2.12/com/nvidia/spark/rapids/GpuBaseAggregateHelper.scala

sql-plugin/src/main/spark340/scala/com/nvidia/spark/rapids/shuffle/RapidsShuffleIterator.scala

tgravescs · 2023-10-26T15:11:49Z

CONTRIBUTING.md

@@ -60,6 +60,9 @@ You can find all available build versions in the top level pom.xml file. If you
 for Databricks then you should use the `jenkins/databricks/build.sh` script and modify it for
 the version you want.

+See the [scala2.13](scala2.13) directory for instructions on how to build against 


might also be nice to state that the code must compile against both versions so when developing keep that in mind. Not sure if its useful to mention a few common things that arise. Also might point to the 2.13/2.12 source subdirectories

Added some documentation updates, please let me know if those make sense.

sql-plugin/src/main/scala-2.12/com/nvidia/spark/rapids/ArmScalaFixes.scala

sql-plugin/src/main/scala-2.13/com/nvidia/spark/rapids/ArmScalaFixes.scala

sql-plugin/src/main/scala-2.13/com/nvidia/spark/rapids/GpuBaseAggregateHelper.scala

sql-plugin/src/main/scala-2.13/com/nvidia/spark/rapids/GpuSorter.scala

…the Stack implementation Signed-off-by: Navin Kumar <[email protected]>

Signed-off-by: Navin Kumar <[email protected]>

NVnavkumar · 2023-10-26T17:23:47Z

build

Signed-off-by: Navin Kumar <[email protected]>

NVnavkumar · 2023-10-26T17:57:51Z

build

Signed-off-by: Navin Kumar <[email protected]>

NVnavkumar · 2023-10-26T19:40:34Z

build

Signed-off-by: Navin Kumar <[email protected]>

NVnavkumar · 2023-10-26T20:43:53Z

build

tgravescs

overall looks fine, please file the followups from the comments

pxLi · 2023-10-27T01:16:47Z

build

NVnavkumar added 4 commits June 20, 2023 20:11

Preliminary Scala 2.13 build support, need to fix compilation errors

77e83c3

Signed-off-by: Navin Kumar <[email protected]>

Handle 2 simple updates with scalastyle checks

7c9a930

Update due to requirement of not importing scala.collection.Seq directly

5b41add

One round of code updates for Scala 2.13

02ef278

Signed-off-by: Navin Kumar <[email protected]>

gerashegalov reviewed Jun 21, 2023

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

gerashegalov reviewed Jun 21, 2023

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

pxLi reviewed Jun 24, 2023

View reviewed changes

dev/change-scala-version.sh Outdated Show resolved Hide resolved

pxLi reviewed Jun 26, 2023

View reviewed changes

sameerz changed the base branch from branch-23.08 to branch-23.10 August 10, 2023 22:06

sameerz added the feature request New feature or request label Aug 14, 2023

NVnavkumar added 13 commits September 5, 2023 11:10

Merge branch 'branch-23.10' into scala-213

8adb72e

Style fixes.

ef6a97d

Update RAPIDS shuffle manager code to parallel use of scala.collectio…

a359bec

…n.Seq from Spark 3.4 and higher. This is to avoid unnecessary copying of data.

Handle usage of Seq in GpuBindReferences

65e3601

Sub partition hash join updates. Follow the recommended scala pattern…

bc5d337

… of converting when returning Seq and using Seq in _* pattern

Handle ArrayStack usage in GpuSortExec and SortUtils

ce9054e

Regex parser fixes.

272725e

Parquet Scan Seq usage.

cd7a132

Orc scan Seq usage.

ac75ac6

Multi file reader Seq usage.

c5f7bff

Hive CTAS command fixes

4b69ec9

miscellaneous scan usage of Seq

4714ca6

Fix procedure syntax.

85c0af9

NVnavkumar force-pushed the scala-213 branch from 53a44aa to 85c0af9 Compare September 7, 2023 00:04

NVnavkumar added 4 commits September 6, 2023 17:04

Add procedure declaration checker since it's fully deprecated in Scal…

144e442

…a 2.13 and will cause compilation to fail Signed-off-by: Navin Kumar <[email protected]>

Add scala versions here based on Spark

a38a91a

Merge branch 'branch-23.10' into scala-213

a9bd68e

Move these files to scala 2.12 only compilation

8db9a16

Remove checking scala2.13 folder

c88320b

Remove checking scala2.13 folder, .github/workflows/mvn-verify-check.yml will check it Signed-off-by: Tim Liu <[email protected]>

NvTimLiu dismissed their stale review via c88320b October 26, 2023 03:48

fix typo

5db5520

NvTimLiu previously approved these changes Oct 26, 2023

View reviewed changes

tgravescs reviewed Oct 26, 2023

View reviewed changes

scala2.13/aggregator/pom.xml Show resolved Hide resolved

tgravescs reviewed Oct 26, 2023

View reviewed changes

abellina reviewed Oct 26, 2023

View reviewed changes

sql-plugin/src/main/scala-2.12/com/nvidia/spark/rapids/ArmScalaFixes.scala Outdated Show resolved Hide resolved

abellina reviewed Oct 26, 2023

View reviewed changes

sql-plugin/src/main/scala-2.13/com/nvidia/spark/rapids/ArmScalaFixes.scala Outdated Show resolved Hide resolved

abellina reviewed Oct 26, 2023

View reviewed changes

sql-plugin/src/main/scala-2.13/com/nvidia/spark/rapids/GpuBaseAggregateHelper.scala Outdated Show resolved Hide resolved

abellina reviewed Oct 26, 2023

View reviewed changes

sql-plugin/src/main/scala-2.13/com/nvidia/spark/rapids/GpuSorter.scala Outdated Show resolved Hide resolved

Refactor GpuSorter further to keep it in SortUtils and just shim out …

eb2dbd3

…the Stack implementation Signed-off-by: Navin Kumar <[email protected]>

NVnavkumar dismissed NvTimLiu’s stale review via eb2dbd3 October 26, 2023 17:06

remove commented out code and add comment for clarification

b375ba3

Signed-off-by: Navin Kumar <[email protected]>

Fix issue with copyrights

e6409c8

Signed-off-by: Navin Kumar <[email protected]>

use asIsntanceOf[...] to resolve this Scala 2.13/2.12 discrepancy

b1e774c

Signed-off-by: Navin Kumar <[email protected]>

NVnavkumar added 3 commits October 26, 2023 12:50

Fix newline end of file

bbb6750

Signed-off-by: Navin Kumar <[email protected]>

Update documentation

4a41342

Signed-off-by: Navin Kumar <[email protected]>

Update documentation 2

e0ff07c

Signed-off-by: Navin Kumar <[email protected]>

tgravescs approved these changes Oct 26, 2023

View reviewed changes

This was referenced Oct 26, 2023

[BUG] Scala 2.13 build with JDK 11 or 17 fails OpcodeSuite tests #9555

Closed

[DOC] Add documentation for working with VSCode + Scala Metals extension and Scala 2.13 development #9556

Open

NVnavkumar merged commit b5891fc into NVIDIA:branch-23.12 Oct 27, 2023
36 of 37 checks passed

NVnavkumar mentioned this pull request Oct 27, 2023

[FEA] Scala 2.13 build should emit an error message when trying to build with unsupported shim #9563

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scala 2.13 Support #8592

Scala 2.13 Support #8592

NVnavkumar commented Jun 21, 2023 •

edited

Loading

pxLi left a comment •

edited

Loading

sameerz commented Jun 27, 2023

pxLi commented Jun 27, 2023

NvTimLiu commented Oct 26, 2023 •

edited

Loading

NvTimLiu commented Oct 26, 2023

NvTimLiu left a comment

tgravescs Oct 26, 2023

tgravescs Oct 26, 2023

NVnavkumar Oct 26, 2023

NVnavkumar commented Oct 26, 2023

NVnavkumar commented Oct 26, 2023

NVnavkumar commented Oct 26, 2023

NVnavkumar commented Oct 26, 2023

tgravescs left a comment

pxLi commented Oct 27, 2023


		That way any new dependencies or other changes will be picked up in the Scala 2.13 build.

		## IDE Integration with IntelliJ

Scala 2.13 Support #8592

Scala 2.13 Support #8592

Conversation

NVnavkumar commented Jun 21, 2023 • edited Loading

pxLi left a comment • edited Loading

Choose a reason for hiding this comment

sameerz commented Jun 27, 2023

pxLi commented Jun 27, 2023

NvTimLiu commented Oct 26, 2023 • edited Loading

NvTimLiu commented Oct 26, 2023

NvTimLiu left a comment

Choose a reason for hiding this comment

tgravescs Oct 26, 2023

Choose a reason for hiding this comment

tgravescs Oct 26, 2023

Choose a reason for hiding this comment

NVnavkumar Oct 26, 2023

Choose a reason for hiding this comment

NVnavkumar commented Oct 26, 2023

NVnavkumar commented Oct 26, 2023

NVnavkumar commented Oct 26, 2023

NVnavkumar commented Oct 26, 2023

tgravescs left a comment

Choose a reason for hiding this comment

pxLi commented Oct 27, 2023

NVnavkumar commented Jun 21, 2023 •

edited

Loading

pxLi left a comment •

edited

Loading

NvTimLiu commented Oct 26, 2023 •

edited

Loading