[BUG] Nightly build failed: ExecutionPlanCaptureCallback$.class is not bitwise-identical across shims

[tgravescs]

Describe the bug
exec binary-dedupe.sh failed, exit code is 255, error msg is org/apache/spark/sql/rapids/ExecutionPlanCaptureCallback$.class is not bitwise-identical across shims

There was a change yesterday to this file, so perhaps its pulling in something that isn't shim agnostic:
[[https://github.com/NVIDIA/spark-rapids/commit/1fb5fc4cbea646588fababca5c50a8829e90348e#diff-c2b3f7332a76954f1e54afe7[…]707057b51d20faddd33db7b0045fd](https://github.com/NVIDIA/spark-rapids/commit/1fb5fc4cbea646588fababca5c50a8829e90348e#diff-c2b3f7332a76954f1e54afe7a088136071e707057b51d20faddd33db7b0045fd)](1fb5fc4#diff-c2b3f7332a76954f1e54afe7a088136071e707057b51d20faddd33db7b0045fd)

[mythrocks]

perhaps its pulling in something that isn't shim agnostic...

At first glance, I don't see what it might be.

This test pulls in the following:

import org.apache.spark.sql.execution.adaptive.{..., AdaptiveSparkPlanHelper, ...}
...
import org.apache.spark.sql.execution.FileSourceScanExec
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser

Judging from how these classes are used elsewhere, it appears that they don't need shimming. This needs a closer look.

Edit: I wonder if it's because GpuFileSourceScanExec is being used here. It has dependencies on some shimmed classes.

import com.nvidia.spark.rapids.shims.{GpuDataSourceRDD, PartitionedFileUtilsShim, SparkShimImpl}

[gerashegalov]

Looking at it with @razajafri, it appears to be caused by extends AdaptiveSparkPlanHelper that pulls more methods on the DBR side. This could be even a compatible difference because methods are added and some method signatures are changed to receive more params but they have a default value. However, our current logic requires bitwise identity for a fool-proof compatibility. So this trait should be inherited via a shim such as PlanShims or a new dedicated one.

[jlowe]

Does this need to target branch-23.08 or is it only an issue in 23.10?

[gerashegalov]

Does this need to target branch-23.08 or is it only an issue in 23.10?

We should target 23.08 because that what the offending PR's target is https://github.com/NVIDIA/spark-rapids/pull/8744/files#diff-c2b3f7332a76954f1e54afe7a088136071e707057b51d20faddd33db7b0045fdR32

It passed CI because pre-merge did not include [databricks]

[sameerz]

Retargeting this issue to 23.10, as the original PR is a test case and is reverted in 23.08.

[jlowe]

This should not have been retargeted but rather simply closed. The bug is fixed, it just happened to be fixed by reverting rather than a new code change.