[BUG] Documented deployment of spark-avro is not tested

[gerashegalov]

Describe the bug
The documented way of deploying spark-avro which we are likely to encounter in the field is by having Spark load it at run time via --packages option from Maven.

However CI only tests with static classpath using spark.*.extraClassPath option.

Steps/Code to reproduce bug
run intergration tests with -k avro with TEST_PARALLEL > 2

Expected behavior
Prioritize testing the advertised way of deploying --packages spark-avro if we need to pick one.

Environment details (please complete the following information)
CI

Additional context
#5648

[pxLi]

This sounds more like spark thing to me instead of our plugin one?

--packages                  Comma-separated list of maven coordinates of jars to include
                              on the driver and executor classpaths. Will search the local
                              maven repo, then maven central and any additional remote
                              repositories given by --repositories. The format for the
                              coordinates should be groupId:artifactId:version.

Even it failed, we should file ticket to spark :)

Also can you provide the detailed --packages=xxxx, like which avro pkg are we expecting here.
And our https://github.com/NVIDIA/spark-rapids/blob/branch-22.08/integration_tests/run_pyspark_from_build.sh does not support packages deployment yet, we may need to do some refactor. Do we really need to cover this one?

Thanks!

[GaryShen2008]

Hi @gerashegalov,
Do we need to run all the avro test cases in integration test to check the --packages deploy way for Avro reading?
If not, I wonder if we just set up a CI job to run a spark job with Avro reading.
What's the point of using TEST_PARALLEL > 2?

[gerashegalov]

This sounds more like spark thing to me instead of our plugin one?

Even it failed, we should file ticket to spark :)
Not really.

This would be the case only for bugs that reproduce even without our Plugin activated. However, our plugin has code affected by this feature and it has exhibited bugs unique to our plugin.

We obviously file and fix upstream issues when appropriate.

The goal of this issue to proactively discover the issues our users are likely to hit before they do and come up with workarounds, fixes for the Plugin and upstream Spark when necessary.

Also can you provide the detailed --packages=xxxx, like which avro pkg are we expecting here. And our https://github.com/NVIDIA/spark-rapids/blob/branch-22.08/integration_tests/run_pyspark_from_build.sh does not support packages deployment yet, we may need to do some refactor. Do we really need to cover this one?

It's documented in the upstream spark-avro doc link provided in the issue description

org.apache.spark:spark-avro_2.12:sparkVersion

[gerashegalov]

Hi @gerashegalov, Do we need to run all the avro test cases in integration test to check the --packages deploy way for Avro reading?

it's always a matter how hard you want to think if you missed anything. So the usual trade-off of test coverage vs resource spend on testing.

If not, I wonder if we just set up a CI job to run a spark job with Avro reading.

yes, this is a minimum viable solution, again a matter of coverage whether it will have all the elements we cover in integration tests. IMO factoring out the Avro integration tests into a dedicated sub-pipeline paramerized on deployment options would be ideal. An MVP solution is a good start though.

What's the point of using TEST_PARALLEL > 2?

This is an interesting artifact tracked in #5714 . We deploy jars differently based on TEST_PARALLEL

[pxLi]

@gerashegalov @sameerz as iceberg test include --packages method https://github.com/NVIDIA/spark-rapids/blob/branch-22.08/jenkins/spark-tests.sh#L187
which covers some sanity check for this issue, do you guys mind we move the ticket to 22.10 for now?

[sameerz]

I am ok to move this to 22.10.

[gerashegalov]

I am fine moving it, too, because it's not a regression.

[pxLi]

thanks! @zhanga5 will help look into this later