Skip to content

Commit

Permalink
Switch lightweight-spark-distrib to the VL fork & bump to 0.0.5 (#…
Browse files Browse the repository at this point in the history
  • Loading branch information
Gedochao authored Nov 6, 2023
1 parent eced0b3 commit bb68631
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 3 deletions.
2 changes: 1 addition & 1 deletion DEV.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ Here's some of the more important external projects used by Scala CLI:
stripped up of its benchmark infrastructure and build integrations.
- [no-crc32-zip-input-stream](https://github.com/scala-cli/no-crc32-zip-input-stream): A copy of `ZipInputStream`
from OpenJDK, with CRC32 calculations disabled.
- [lightweight-spark-distrib](https://github.com/scala-cli/lightweight-spark-distrib): a small application allowing
- [lightweight-spark-distrib](https://github.com/VirtusLab/lightweight-spark-distrib): a small application allowing
to make Spark distributions more lightweight.
- [java-class-name](https://github.com/scala-cli/java-class-name): a small library to extract class names
from Java sources.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ import scala.util.Properties

object SparkTestDefinitions {

def lightweightSparkDistribVersionOpt = Option("0.0.4")
def lightweightSparkDistribVersionOpt: Option[String] = Option("0.0.5")

final class Spark(val sparkVersion: String, val scalaVersion: String) {
private def sbv = scalaVersion.split('.').take(2).mkString(".")
private var toDeleteOpt = Option.empty[os.Path]
lazy val sparkHome: os.Path = {
val url = lightweightSparkDistribVersionOpt match {
case Some(lightweightSparkDistribVersion) =>
s"https://github.com/scala-cli/lightweight-spark-distrib/releases/download/v$lightweightSparkDistribVersion/spark-$sparkVersion-bin-hadoop2.7-scala$sbv.tgz"
s"https://github.com/VirtusLab/lightweight-spark-distrib/releases/download/v$lightweightSparkDistribVersion/spark-$sparkVersion-bin-hadoop2.7-scala$sbv.tgz"
case None =>
// original URL (too heavyweight, often fails / times out…)
s"https://archive.apache.org/dist/spark/spark-$sparkVersion/spark-$sparkVersion-bin-hadoop2.7.tgz"
Expand Down

0 comments on commit bb68631

Please sign in to comment.