Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] MatchError in getDataSourceInfo #1042

Closed
amahussein opened this issue May 28, 2024 · 1 comment · Fixed by #1052
Closed

[BUG] MatchError in getDataSourceInfo #1042

amahussein opened this issue May 28, 2024 · 1 comment · Fixed by #1052
Assignees
Labels
bug Something isn't working core_tools Scope the core module (scala)

Comments

@amahussein
Copy link
Collaborator

amahussein commented May 28, 2024

Describe the bug

The Profiler V24.02.2 throws an error while processing the dataSourceInfo

24/05/24 15:55:14 WARN Profiler: Exception occurred processing file: application_11111111111_0001_1
scala.MatchError: scan time total (min, med, max) (of class java.lang.String)
        at com.nvidia.spark.rapids.tool.profiling.CollectInformation.$anonfun$getDataSourceInfo$2(CollectInformation.scala:108)
        at com.nvidia.spark.rapids.tool.profiling.CollectInformation.$anonfun$getDataSourceInfo$2$adapted(CollectInformation.scala:108)
        at scala.collection.immutable.List.map(List.scala:293)
        at com.nvidia.spark.rapids.tool.profiling.CollectInformation.getIoMetrics$1(CollectInformation.scala:108)
        at com.nvidia.spark.rapids.tool.profiling.CollectInformation.$anonfun$getDataSourceInfo$6(CollectInformation.scala:129)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at scala.collection.TraversableLike.map(TraversableLike.scala:286)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at com.nvidia.spark.rapids.tool.profiling.CollectInformation.$anonfun$getDataSourceInfo$3(CollectInformation.scala:125)
        at scala.collection.immutable.List.flatMap(List.scala:366)
        at com.nvidia.spark.rapids.tool.profiling.CollectInformation.getDataSourceInfo(CollectInformation.scala:117)
        at com.nvidia.spark.rapids.tool.profiling.Profiler.com$nvidia$spark$rapids$tool$profiling$Profiler$$processApps(Profiler.scala:340)
        at com.nvidia.spark.rapids.tool.profiling.Profiler$ProfileProcessThread$1.run(Profiler.scala:269)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
@amahussein amahussein added bug Something isn't working core_tools Scope the core module (scala) labels May 28, 2024
@amahussein amahussein self-assigned this May 28, 2024
@amahussein
Copy link
Collaborator Author

This looks to be caused by and old spark version eventlog 2.3.4.
We may need to figure out how to handle the metrics for those old spark eventlogs.

amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue May 30, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Fixes NVIDIA#1042
amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue May 30, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Fixes NVIDIA#1042
amahussein added a commit that referenced this issue May 31, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Fixes #1042
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working core_tools Scope the core module (scala)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant