Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace anonymous classes for SortOrder and FIlterExec overrides #10839

Merged
merged 11 commits into from
May 23, 2024

Conversation

gerashegalov
Copy link
Collaborator

@gerashegalov gerashegalov commented May 18, 2024

Demo PR contributing to #10838

It showcases a coding convention to follow using SortOrder and FilterExec replacements as an example

scala>  spark.range(100).where($"id" <= 10).collect()

java.lang.RuntimeException: convertToGpu failed
  at scala.sys.package$.error(package.scala:30)
  at com.nvidia.spark.rapids.GpuFilterExecMeta.convertToGpu(basicPhysicalOperators.scala:790)
  at com.nvidia.spark.rapids.GpuFilterExecMeta.convertToGpu(basicPhysicalOperators.scala:783)
  at com.nvidia.spark.rapids.SparkPlanMeta.convertIfNeeded(RapidsMeta.scala:838)
  at com.nvidia.spark.rapids.GpuOverrides$.com$nvidia$spark$rapids$GpuOverrides$$doConvertPlan(GpuOverrides.scala:4383)
  at com.nvidia.spark.rapids.GpuOverrides.applyOverrides(GpuOverrides.scala:4728)
  at com.nvidia.spark.rapids.GpuOverrides.$anonfun$applyWithContext$3(GpuOverrides.scala:4588)
  at com.nvidia.spark.rapids.GpuOverrides$.logDuration(GpuOverrides.scala:455)
  at com.nvidia.spark.rapids.GpuOverrides.$anonfun$applyWithContext$1(GpuOverrides.scala:4585)
  at com.nvidia.spark.rapids.GpuOverrideUtil$.$anonfun$tryOverride$1(GpuOverrides.scala:4551)
  at com.nvidia.spark.rapids.GpuOverrides.applyWithContext(GpuOverrides.scala:4605)
  at com.nvidia.spark.rapids.GpuOverrides.apply(GpuOverrides.scala:4578)
  at com.nvidia.spark.rapids.GpuOverrides.apply(GpuOverrides.scala:4574)
  at org.apache.spark.sql.execution.ApplyColumnarRulesAndInsertTransitions.$anonfun$apply$1(Columnar.scala:532)

Signed-off-by: Gera Shegalov [email protected]

node being replaced for better diagnostics

Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov gerashegalov changed the title [WIP] GpuOverrides exception should be tied to SparkPlan node being replaced for better diagnostics Replace anonymous classes for SortOrder and FIlterExec overrides May 22, 2024
@gerashegalov gerashegalov self-assigned this May 22, 2024
@gerashegalov gerashegalov marked this pull request as ready for review May 22, 2024 05:32
@gerashegalov
Copy link
Collaborator Author

build

Signed-off-by: Gera Shegalov <[email protected]>
Copy link
Member

@jlowe jlowe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the anonymous class changes, since we're trading class-for-class, making the stacktrace much more useful without adding to the classloading burden. As for the pluginChecks change, that I'm less convinced is a net benefit change.

Signed-off-by: Gera Shegalov <[email protected]>
Signed-off-by: Gera Shegalov <[email protected]>
Signed-off-by: Gera Shegalov <[email protected]>
Signed-off-by: Gera Shegalov <[email protected]>
Signed-off-by: Gera Shegalov <[email protected]>
Signed-off-by: Gera Shegalov <[email protected]>
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
Copy link
Collaborator Author

build

@gerashegalov gerashegalov merged commit 1a9b345 into NVIDIA:branch-24.06 May 23, 2024
44 checks passed
@gerashegalov gerashegalov deleted the unbloatGpuOverrides branch May 23, 2024 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants