UnSupported Spark Version: 3.0.2

[ghost]

It seems DotnetRunner class does not support spark versions above 3.0.1. Is it possible to disable version checking somehow? i have spark 3.0.2 and looking at the scala source code, it seems the max supported version is 3.0.1. Is there a workaround for this? Or can you guys update this for the latest Spark versions?

Exception in thread "main" java.lang.IllegalArgumentException: Unsupported spark version used: 3.0.2. Normalized spark version used: 3.0.2. Supported versions: 3.0.0, 3.0.1
        at org.apache.spark.deploy.dotnet.DotnetRunner$.validateSparkVersions(DotnetRunner.scala:172)

Thanks
Ali

[imback82]

Thank @ali-khawaja for reporting this. Supporting 3.0.2 (#833) will come in the next release.

But we will try to make the experience better and introduce a config to disable the check.

[ghost]

and btw it gets stuck. you have to do ctrl+c to get back the prompt.

[ghost]

i upgraded spark to 3.1.1, rebuild the spark.dotnet, and copied the newly built jar and it still gave the same error message as above: nullpointerexception.

[ghost]

sparkSparkDriver.log

I have enabled log4j DEBUG and attached the detailed logs as well

[imback82]

21/03/18 23:27:27 WARN AbstractChannelHandlerContext: An exception 'java.lang.NullPointerException' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.io.EOFException
at java.base/java.io.DataInputStream.readFully(DataInputStream.java:202)
at java.base/java.io.DataInputStream.readFully(DataInputStream.java:170)
at org.apache.spark.api.dotnet.SerDe.readStringBytes(SerDe.scala:73)
at org.apache.spark.api.dotnet.SerDe.readString(SerDe.scala:80)

Look at this stack, it seems that bad bytes are being passed from C# to JVM (possible cause for OOM if the number of bytes to read is long). Can you try compiling C# code (Microsoft.Spark.dll) from the source as well and use it?

[ghost]

thanks. i rebuild csharp project as well (had to update the project files for .net core 3.1) . it fixed the problem. Really appreciate it.

akhawaja@eluktronics-ak:~/git/sparkdotnetcode/sparkapp01$ ./run.sh
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Dotnet Word Count Example
/home/akhawaja/git/sparkdotnetcode/sparkapp01
[2021-03-20T06:17:31.5925473Z] [eluktronics-ak] [Info] [ConfigurationService] Using port 35441 for connection.
[2021-03-20T06:17:31.5960007Z] [eluktronics-ak] [Info] [JvmBridge] JvMBridge port is 35441
[2021-03-20T06:17:31.5983538Z] [eluktronics-ak] [Info] [JvmBridge] The number of JVM backend thread is set to 10. The max number of concurrent sockets in JvmBridge is set to 7.
Microsoft.Spark.SparkContext
+------+-----+
|  word|count|
+------+-----+
|  .NET|    3|
|  This|    2|
|   app|    2|
|Apache|    2|
| Spark|    2|
|   for|    1|
|  uses|    1|
|  with|    1|
|counts|    1|
| Hello|    1|
| World|    1|
| words|    1|
+------+-----+

akhawaja@eluktronics-ak:~/git/sparkdotnetcode/sparkapp01$