No Encoder found for Double

[ToddSmall]

Hi,

I've modified the example in the Quick Start Guide to use the following main function:

@file:JvmName("SimpleApp")
import org.jetbrains.spark.api.*

data class LonLat(val lon: Double, val lat: Double)

fun main() {
    withSpark {
        spark.dsOf(LonLat(1.0, 2.0), LonLat(3.0, 4.0)).show()
    }
}

When I execute this program with:

spark-3.0.0-bin-hadoop2.7/bin/spark-submit --class "SimpleApp" --master local src/IdeaProjects/kotlin-spark-example/build/libs/kotlin-spark-example-1.0-SNAPSHOT-all.jar

I get the following error:

Exception in thread "main" java.lang.UnsupportedOperationException: No Encoder found for Double
- field (class: "scala.Double", name: "lat")
- root class: "LonLat"
	at org.apache.spark.sql.KotlinReflection$.$anonfun$deserializerFor$1(KotlinReflection.scala:426)
	at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:69)
	at org.apache.spark.sql.KotlinReflection.cleanUpReflectionObjects(KotlinReflection.scala:864)
	at org.apache.spark.sql.KotlinReflection.cleanUpReflectionObjects$(KotlinReflection.scala:863)
	at org.apache.spark.sql.KotlinReflection$.cleanUpReflectionObjects(KotlinReflection.scala:47)
	at org.apache.spark.sql.KotlinReflection$.deserializerFor(KotlinReflection.scala:202)
	at org.apache.spark.sql.KotlinReflection$.$anonfun$deserializerFor$7(KotlinReflection.scala:351)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
	at scala.collection.TraversableLike.map(TraversableLike.scala:238)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
	at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
	at org.apache.spark.sql.KotlinReflection$.$anonfun$deserializerFor$1(KotlinReflection.scala:340)
	at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:69)
	at org.apache.spark.sql.KotlinReflection.cleanUpReflectionObjects(KotlinReflection.scala:864)
	at org.apache.spark.sql.KotlinReflection.cleanUpReflectionObjects$(KotlinReflection.scala:863)
	at org.apache.spark.sql.KotlinReflection$.cleanUpReflectionObjects(KotlinReflection.scala:47)
	at org.apache.spark.sql.KotlinReflection$.deserializerFor(KotlinReflection.scala:202)
	at org.apache.spark.sql.KotlinReflection$.$anonfun$deserializerFor$20(KotlinReflection.scala:470)
	at org.apache.spark.sql.catalyst.DeserializerBuildHelper$.deserializerForWithNullSafetyAndUpcast(DeserializerBuildHelper.scala:54)
	at org.apache.spark.sql.KotlinReflection$.deserializerFor(KotlinReflection.scala:470)
	at org.apache.spark.sql.KotlinReflection.deserializerFor(KotlinReflection.scala)
	at org.jetbrains.spark.api.ApiV1Kt.kotlinClassEncoder(ApiV1.kt:103)
	at org.jetbrains.spark.api.ApiV1Kt.generateEncoder(ApiV1.kt:91)
	at SimpleApp.main(SimpleApp.kt:47)
	at SimpleApp.main(SimpleApp.kt)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

If I replace the Doubles in the data class with Ints, then the program executes correctly, printing:

+---+---+
|lat|lon|
+---+---+
|  2|  1|
|  4|  3|
+---+---+

My build environment is identical to the one at https://github.com/MKhalusova/kotlin-spark-example, and I'm using Spark 3.0.0 with Java 1.8.0_252 (AdoptOpenJDK) on MacOS 10.15.6.

Thank you,

Todd

[ToddSmall]

I just noticed that build.gradle on https://github.com/MKhalusova/kotlin-spark-example references version 0.1.0 of kotlin-spark-api. However, the issue with "No Encoder found for Double" still exists when I use version 0.3.0 of kotlin-spark-api.

Todd

[MKhalusova]

@ToddSmall Thanks for noticing! I'll update the version in the example.

[ToddSmall]

Thank you!

[asm0dey]

@ToddSmall Thank you for the report! Could you test please?

[ToddSmall]

@asm0dey Yes, my issue is fixed. Thanks again!

Todd