Add test for selecting a single complex field array and its parent struct array [databricks]

[razajafri]

Pushing this comprehensive patch after the revert of #8744.

PR tries to mimic the test in Spark's SchemaPruningSuite.

We create a table with complex fields and try to read only a subfield to see if column pruning is working the way it's supposed to i.e. we are not reading unnecessary columns. e.g. if we have a complex type contact with an array of friends, like so

Contact {
   first_name: String
   middle_name: String
   last_name: String
   friends: Array[Contact]
}

selecting spark.table("contacts").select(explode("friends").alias("friend").select("friend.first_name") shouldn't also read the middle_name and last_name of the friends field.

fixes #8712
fixes #8713
fixes #8714
fixes #8715

[razajafri]

Depends on #9013

[razajafri]

I will add comment here stating why this file is needed

[razajafri]

@gerashegalov @jlowe I have tested the jar on DB and the parallel-worlds has two separate files for AdaptiveSparkPlanHelperImpl, do I need to add this filename to any other txt files that we are using in de-dupe process?

[jlowe]

I have tested the jar on DB and the parallel-worlds has two separate files for AdaptiveSparkPlanHelperImpl

That is expected, because AdaptiveSparkPlanHelperImpl derives from AdaptiveSparkPlanHelper which is different across the Spark platforms.

[jlowe]

ShimLoader should be returning an unshimmed class here, otherwise an unshimmed class may attempt to load a shimmed class even before we get to the ShimLoader part, and if the classpath isn't parallel-worlds aware, that will fail.

[gerashegalov]

nit: we can do this already in parametrize

@pytest.mark.parametrize('query,expected_schemata',
def test_select_complex_field(..., query, expected_schemata, ...)

[razajafri]

For some reason, pytest doesn't like it when I mark one of the parameters namely pytest.param(("select name.middle, address from {} where p=2", "struct<name:struct<middle:string>,address:string>"), marks=pytest.mark.skip(reason='https://github.com/NVIDIA/spark-rapids/issues/8788')),

[gerashegalov]

it looks like you are passing a pair to pytest,param, it should be varargs https://docs.pytest.org/en/7.1.x/reference/reference.html#pytest-param

[razajafri]

build

[jlowe]

AdaptiveSparkPlanHelperShim.class needs to be added to dist/unshimmed-common-from-spark311.txt.

[razajafri]

build

[razajafri]

@gerashegalov @jlowe please take another look

[jlowe]

@razajafri any reason this is still a draft PR?

[razajafri]

@razajafri any reason this is still a draft PR?

Thanks for the review.

I had put it in draft because I wanted to ensure the pre-merge passed after @gerashegalov 's changes to the multi-shim jar.

[jlowe]

I had put it in draft because I wanted to ensure the pre-merge passed

Note that if the only reason for the draft is to make sure premerge passed, then IMO there's no reason for the PR to be draft. A failed premerge will prevent it from being merged even if it's not in draft. The main reason for draft is to prevent a PR from being merged even if it passes premerge and otherwise would be eligible for merging.