Scala 2.13 cross compilation

[pomadchin]

Basing on the following information

https://spark.apache.org/news/spark-2-4-1-released.html

In Apache Spark 2.4.1, Scala 2.12 support is GA, and it's no longer experimental. 
We will drop Scala 2.11 support in Spark 3.0, so please provide us feedback.

And this JIRA ticket, I suppose we need to move along with the Spark.

Probably 2.13 support would be in Spark 3.0 release. The result of this issue would be to drop Scala 2.11 and to add Scala 2.12 and 2.13 cross compilation (for sure if there would be Spark binaries).

This issue will also contain changes that would be nice to have to simplify Scala 3 migration:

Scala 2.13

• Remove usage of Traversable (it is possible to replace it with Iterable)
• Remove usage of TraversableOnce (it is possible to replace it with IterableOnce)
• Drop Scala 2.11

Scala 3

• After Scala 2.13 function def foo() can be called only as a .foo() in the code and can't be invoked as .foo without parentheses (see https://scastie.scala-lang.org/pzjfiB1VT1C9Aljp9YkzMQ). So Scala would be more strict about it.
• do loops would be removed; instead, it would be possible to use a new while notation
• Class shadowing
• XML literals
• Symbol literals
• Weak Conformance
• private[this] and protected[this] modifiers

[pomadchin]

• Spark 3.2 snapshots for Scala 2.13 are available for use now, today: https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-core_2.13/
• With any luck, Spark 3.2 final will ship this summer: http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Spark-3-2-Expectation-td30826.html

source: https://twitter.com/SethTisue/status/1372340143529467905?s=20

[jessewebb]

This issue is closed because the MR (#3294) above was merged. However, when I look on Maven Central for the Scala 2.13 versions, it appears they weren't published as part of the 3.6.0 release.

i.e. https://search.maven.org/search?q=org.locationtech.geotrellis
i.e. https://mvnrepository.com/artifact/org.locationtech.geotrellis/geotrellis-vector

Both links above show no 2.13 binaries available.

When can we expect 2.13 binaries to be published?

[pomadchin]

Hey @jessewebb, that's correct. We are blocked by the Spark 3.2.0 release now. I think the ETA for that is ~ September / October. The 2.13 crosscompilation relies on the Spark 3.2.0-SNAPSHOT, because of this reason we publish only 2.13 snapshots that can be found here: https://repo.eclipse.org/content/groups/snapshots/org/locationtech/geotrellis/

[jessewebb]

Thank you for the quick response! We are moving our codebases to 2.13 and we are just using the -vector JAR right now for some Point datatypes and we don't rely on Spark. I think it will be safe for us to use the SNAPSHOT versions for now until you release an official build. Thanks!

[waveskimmer]

With Spark 3.2.0 released in Oct, would it be possible to get a 2.13 build?

[pomadchin]

hey @waveskimmer will publish it soon, for now I would recommend you to use SNAPSHOTS, we don't really plan to do any changes before the release.