Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update spark to 3.3.2 #198

Merged
merged 7 commits into from
Jul 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
strategy:
matrix:
scala: [ "2.12.17", "2.13.10" ]
spark: [ "3.3.1", "3.3.0", "3.2.3", "3.2.2", "3.2.1", "3.2.0", "3.1.3", "3.1.2", "3.1.1", "3.1.0", "3.0.3", "3.0.2", "3.0.1", "3.0.0" ]
spark: [ "3.3.2", "3.3.1", "3.3.0", "3.2.3", "3.2.2", "3.2.1", "3.2.0", "3.1.3", "3.1.2", "3.1.1", "3.1.0", "3.0.3", "3.0.2", "3.0.1", "3.0.0" ]
exclude:
- scala: "2.13.10"
spark: "3.1.3"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish_dev_version.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
strategy:
matrix:
scala: [ "2.12.17", "2.13.10" ]
spark: [ "3.3.1", "3.3.0", "3.2.3", "3.2.2", "3.2.1", "3.2.0", "3.1.3", "3.1.2", "3.1.1", "3.1.0", "3.0.3", "3.0.2", "3.0.1", "3.0.0" ]
spark: [ "3.3.2", "3.3.1", "3.3.0", "3.2.3", "3.2.2", "3.2.1", "3.2.0", "3.1.3", "3.1.2", "3.1.1", "3.1.0", "3.0.3", "3.0.2", "3.0.1", "3.0.0" ]
exclude:
- scala: "2.13.10"
spark: "3.1.3"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish_release_version.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
strategy:
matrix:
scala: [ "2.12.17", "2.13.10" ]
spark: [ "3.3.1", "3.3.0", "3.2.3", "3.2.2", "3.2.1", "3.2.0", "3.1.3", "3.1.2", "3.1.1", "3.1.0", "3.0.3", "3.0.2", "3.0.1", "3.0.0" ]
spark: [ "3.3.2", "3.3.1", "3.3.0", "3.2.3", "3.2.2", "3.2.1", "3.2.0", "3.1.3", "3.1.2", "3.1.1", "3.1.0", "3.0.3", "3.0.2", "3.0.1", "3.0.0" ]
exclude:
- scala: "2.13.10"
spark: "3.1.3"
Expand Down
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

[![Kotlin Stable](https://kotl.in/badges/stable.svg)](https://kotlinlang.org/docs/components-stability.html)
[![JetBrains official project](http://jb.gg/badges/official.svg)](https://confluence.jetbrains.com/display/ALL/JetBrains+on+GitHub)
[![Maven Central](https://img.shields.io/maven-central/v/org.jetbrains.kotlinx.spark/kotlin-spark-api_3.3.1_2.13.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:"org.jetbrains.kotlinx.spark"%20AND%20a:"kotlin-spark-api_3.3.1_2.13")
[![Maven Central](https://img.shields.io/maven-central/v/org.jetbrains.kotlinx.spark/kotlin-spark-api_3.3.2_2.13.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:"org.jetbrains.kotlinx.spark"%20AND%20a:"kotlin-spark-api_3.3.2_2.13")
[![Join the chat at https://gitter.im/JetBrains/kotlin-spark-api](https://badges.gitter.im/JetBrains/kotlin-spark-api.svg)](https://gitter.im/JetBrains/kotlin-spark-api?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

Your next API to work with [Apache Spark](https://spark.apache.org/).
Expand Down Expand Up @@ -38,6 +38,8 @@ We have opened a Spark Project Improvement Proposal: [Kotlin support for Apache

| Apache Spark | Scala | Kotlin for Apache Spark |
|:------------:|:-----:|:-----------------------------------:|
| 3.3.2 | 2.13 | kotlin-spark-api_3.3.2_2.13:VERSION |
| | 2.12 | kotlin-spark-api_3.3.2_2.12:VERSION |
| 3.3.1 | 2.13 | kotlin-spark-api_3.3.1_2.13:VERSION |
| | 2.12 | kotlin-spark-api_3.3.1_2.12:VERSION |
| 3.3.0 | 2.13 | kotlin-spark-api_3.3.0_2.13:VERSION |
Expand Down Expand Up @@ -73,7 +75,7 @@ The Kotlin for Spark artifacts adhere to the following convention:
The only exception to this is `scala-tuples-in-kotlin_[Scala core version]:[Kotlin for Apache Spark API version]`, which is
independent of Spark.

[![Maven Central](https://img.shields.io/maven-central/v/org.jetbrains.kotlinx.spark/kotlin-spark-api_3.3.1_2.13.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:"org.jetbrains.kotlinx.spark"%20AND%20a:"kotlin-spark-api_3.3.1_2.13")
[![Maven Central](https://img.shields.io/maven-central/v/org.jetbrains.kotlinx.spark/kotlin-spark-api_3.3.2_2.13.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:"org.jetbrains.kotlinx.spark"%20AND%20a:"kotlin-spark-api_3.3.2_2.13")

## How to configure Kotlin for Apache Spark in your project

Expand All @@ -84,7 +86,7 @@ Here's an example `pom.xml`:
```xml
<dependency>
<groupId>org.jetbrains.kotlinx.spark</groupId>
<artifactId>kotlin-spark-api_3.3.1_2.13</artifactId>
<artifactId>kotlin-spark-api_3.3.2_2.13</artifactId>
<version>${kotlin-spark-api.version}</version>
</dependency>
<dependency>
Expand Down Expand Up @@ -117,7 +119,7 @@ To it, simply add
to the top of your notebook. This will get the latest version of the API, together with the latest version of Spark.
To define a certain version of Spark or the API itself, simply add it like this:
```jupyterpython
%use spark(spark=3.3.1, scala=2.13, v=1.2.3)
%use spark(spark=3.3.2, scala=2.13, v=1.2.3)
```

Inside the notebook a Spark session will be initiated automatically. This can be accessed via the `spark` value.
Expand Down
18 changes: 9 additions & 9 deletions docs/quick-start-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ Here's what the `pom.xml` looks like for this example:

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<kotlin.version>1.4.0</kotlin.version>
<kotlin.version>1.8.0</kotlin.version>
<kotlin.code.style>official</kotlin.code.style>
</properties>

Expand All @@ -72,13 +72,13 @@ Here's what the `pom.xml` looks like for this example:
</dependency>
<dependency> <!-- Kotlin Spark API dependency -->
<groupId>org.jetbrains.kotlinx.spark</groupId>
<artifactId>kotlin-spark-api_3.3.1_2.13</artifactId>
<artifactId>kotlin-spark-api_3.3.2_2.13</artifactId>
<version>1.2.3</version>
</dependency>
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.3.1</version>
<version>3.3.2</version>
</dependency>
</dependencies>

Expand Down Expand Up @@ -161,10 +161,10 @@ repositories {

dependencies {
// Kotlin stdlib
implementation 'org.jetbrains.kotlin:kotlin-stdlib:1.4.0'
implementation 'org.jetbrains.kotlin:kotlin-stdlib:1.8.0'
// Kotlin Spark API
implementation 'org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.1_2.13:1.2.3' // Apache Spark
compileOnly 'org.apache.spark:spark-sql_2.12:3.3.1'
implementation 'org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.2_2.13:1.2.3' // Apache Spark
compileOnly 'org.apache.spark:spark-sql_2.12:3.3.2'
}

compileKotlin {
Expand All @@ -186,7 +186,7 @@ import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile

plugins {
id ("org.jetbrains.kotlin.jvm") version "1.4.0"
id ("org.jetbrains.kotlin.jvm") version "1.8.0"
id ("com.github.johnrengelman.shadow") version "5.2.0"
}

Expand All @@ -198,9 +198,9 @@ dependencies {
// Kotlin stdlib
implementation ("org.jetbrains.kotlin:kotlin-stdlib:1.4.0")
// Kotlin Spark API
implementation ("org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.1_2.13:1.2.3")
implementation ("org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.2_2.13:1.2.3")
// Apache Spark
compileOnly ("org.apache.spark:spark-sql_2.12:3.3.1")
compileOnly ("org.apache.spark:spark-sql_2.12:3.3.2")
}

compileKotlin.kotlinOptions.jvmTarget = "1.8"
Expand Down
4 changes: 2 additions & 2 deletions gradle.properties
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@ GROUP=org.jetbrains.kotlinx.spark

# Controls the spark and scala version for the entire project
# can also be defined like ./gradlew -Pspark=X.X.X -Pscala=X.X.X build
spark=3.3.1
spark=3.3.2
scala=2.13.10
# scala=2.12.17
skipScalaTuplesInKotlin=false

org.gradle.caching=true
org.gradle.parallel=false
#kotlin.incremental.useClasspathSnapshot=true
#kotlin.incremental.useClasspathSnapshot=true