-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] JDK17 nightly build after Spark UT Framework is merged #10801
Comments
The stacktrace pasted by Tim is expected in The real failed test case is The root cause of this failed test case is that, when Spark-Rapids is built by JDK 17 and Run on JDK 17, the expression I have tried to reproduce this behavior in a simple project (https://github.com/binmahone/test_jdk17_java, you can download it, build & run it with JDK17), the output is always "GroupByType$". Even if I have tried to move many pof Spark-Rapids's pom setttings to the simple project (such as scalatest configurations, etc.), it's still not reproducing the wrong value "". So I have no clue what settings in Spark-Rapids leads to this wrong behaviour. Since this issue looks not very urgent, I'll first ignore the failed test case and unblock Nightly build |
Signed-off-by: Hongbin Ma (Mahone) <[email protected]>
Signed-off-by: Hongbin Ma (Mahone) <[email protected]>
It looks related to https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8198818 but I cannot reproduce it outside Spark and Plugin either. |
hi @gerashegalov, dead link on my side, can you check it please? Also, just to clarify, the behavior is correct in vanilla spark, but wrong in spark-rapids |
@binmahone I can still click through it, maybe it was temporarily unavailable? Can you access and search https://bugs.java.com/bugdatabase/ for "JDK-8198818 : Class.simpleName different for anonymous classes"? It came up in https://youtrack.jetbrains.com/issue/KT-23072/Class.simpleName-of-anonymous-object-is-not-an-empty-string-in-JDK8-but-on-JDK9 I am not sure that Spark test code is actually robust. Calling getSimpleName on weird Scala object might be sub-optimal. |
It turns out, there is a history of getSimpleName being broken with various combinations of Scala version and JDK Yet it is still in use in various places. We just got unlucky that we are testing Spark 3.3 that bundles 2.12.15 . This combo yields an empty string for GroupByType $ echo 'Seq(1,2,3).toDF.groupBy($"value")' | JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 /home/gshegalov/dist/spark-3.3.4-bin-hadoop2/bin/spark-shell |& grep 'Scala\|Relational'
Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 17.0.10)
res0: org.apache.spark.sql.RelationalGroupedDataset = RelationalGroupedDataset: [grouping expressions: [value], value: [value: int], type: ] And it is fixed in 3.4+ simply because it upgraded to Scala 2.12.17 $ echo 'Seq(1,2,3).toDF.groupBy($"value")' | JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 /home/gshegalov/dist/spark-3.4.0-bin-hadoop3/bin/spark-shell |& grep 'Scala\|Relational'
Using Scala version 2.12.17 (OpenJDK 64-Bit Server VM, Java 17.0.10)
res0: org.apache.spark.sql.RelationalGroupedDataset = RelationalGroupedDataset: [grouping expressions: [value], value: [value: int], type: GroupBy] |
@gerashegalov great findings! It's interesting that you can reproduce the problem with Spark 3.3 + scala 2.12.15 (I tried your way, it reproduces, too), because :
/usr/lib/jvm/java-1.17.0-openjdk-amd64/bin/java -javaagent:/home/hongbin/.local/share/JetBrains/Toolbox/apps/intellij-idea-ultimate/lib/idea_rt.jar=46767:/home/hongbin/.local/share/JetBrains/Toolbox/apps/intellij-idea-ultimate/bin -Dfile.encoding=UTF-8 -classpath /home/hongbin/.local/share/JetBrains/IntelliJIdea2024.1/Scala/lib/runners.jar:/home/hongbin/code/test_jdk17_java/target/spark311/test-classes:/home/hongbin/code/test_jdk17_java/target/spark311/classes:/home/hongbin/.m2/repository/org/scala-lang/scala-library/2.12.15/scala-library-2.12.15.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest_2.12/3.2.16/scalatest_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-core_2.12/3.2.16/scalatest-core_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-compatible/3.2.16/scalatest-compatible-3.2.16.jar:/home/hongbin/.m2/repository/org/scalactic/scalactic_2.12/3.2.16/scalactic_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scala-lang/modules/scala-xml_2.12/2.1.0/scala-xml_2.12-2.1.0.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-featurespec_2.12/3.2.16/scalatest-featurespec_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-flatspec_2.12/3.2.16/scalatest-flatspec_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-freespec_2.12/3.2.16/scalatest-freespec_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-funsuite_2.12/3.2.16/scalatest-funsuite_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-funspec_2.12/3.2.16/scalatest-funspec_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-propspec_2.12/3.2.16/scalatest-propspec_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-refspec_2.12/3.2.16/scalatest-refspec_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-wordspec_2.12/3.2.16/scalatest-wordspec_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-diagrams_2.12/3.2.16/scalatest-diagrams_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-matchers-core_2.12/3.2.16/scalatest-matchers-core_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-shouldmatchers_2.12/3.2.16/scalatest-shouldmatchers_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scalatest/scalatest-mustmatchers_2.12/3.2.16/scalatest-mustmatchers_2.12-3.2.16.jar:/home/hongbin/.m2/repository/org/scala-lang/scala-reflect/2.12.17/scala-reflect-2.12.17.jar org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner -s org.example.SimpleSuite -showProgressMessages true
Testing started at 11:26 am ...
Hello world!
GroupByType$
Even if I manually replace |
So what would be you advice next ? I suggest holding upgrading 2.12.17 until we have more stronger reasons to do so. For the meanwhile we'll still exclude this test case as I did in #10820 |
Keep enabling the UT for all Spark 3.3+ version. Make sure we can make exclude/xfaill tests conditionally just like we do with pytests and "xfail" this particular test only for 3.3.x while keeping it running for 3.4+ Regarding not being able to repro with the toy project, it may have some mismatch with bytecode generation in Spark build:
|
hi @NvTimLiu , with above discussion I think we can close this issue or move this to backlog, what you think? |
Sounds good to me |
I'm confused how this was resolved. @gerashegalov proposed keeping the test and xfailing it only on Spark 3.3 but run it on Spark 3.4+, so we're at least running it somewhere. As it is now, this test was turned off for all Spark versions in #10820 which is not the state I thought we wanted to leave it in. |
If there's followup work to do, there needs to be a way to track that work. Github issues is how we prefer to track. There's stuff to do here, but no issue to track it. Therefore it is very likely it will never be done, because we'll forget that we were supposed to do it. The "KNOWN_ISSUE" points to a closed issue, so either:
|
As advised by @jlowe , I reopened this issue to keep this issue still in track |
Low priority it since it's a conflict between scala 2.12.15 and JDK17. Change it target in 24.08.
|
Describe the bug
UT tests failed on : The value '**' of the type "STRING" cannot be cast to "INT" because it is malformed.
Note: Currently this issue only appeared on spark-rapids nightly build against
JDK17
;Nightly build UT against
JDK11
andJDK8
have not reported these failuresThe text was updated successfully, but these errors were encountered: