Test parquet predicate pushdown for basic types and fields having dots in names [databricks]

[thirtiseven]

Closes #9127
Closes #9094

This PR adds some tests to test parquet predicate pushdown for basic types and fields with dots in names. And also a follow on to orc's ppd test, using `assume' instead of commenting out failed cases.

For PPD testing, these cases are tested:

• CPU write GPU read, GPU write CPU read, and GPU write GPU read.
• Range partitioning on write and predicate is out of DataFrame range, if applicable.
• Data source V1 and V2.

Some contexts: #9119

[thirtiseven]

build

[res-life]

val TEST_CONF = conf("spark.rapids.sql.test.enabled")
   .doc("Intended to be used by unit tests, if enabled all operations must run on the " +
     "GPU or an error happens.")
   .internal()
   .booleanConf
   .createWithDefault(false)

If enable GPU, we should set this as true. Or we may enconter this kind of case:
The test passed, but it's running on CPU silently, and we expect it's running on GPU.

Changes would like:

if (writeGpu || readGpu) {
  spark.rapids.sql.test.enabled = true
}

[revans2]

Actually I think it should be

"spark.rapdis.slq.test.enabled" -> (!writeGpu).toString

or else if we are reading on the GPU, but writing on the CPU we will run into a problem where the we get an error for having things not be on the GPU.

[thirtiseven]

Updated to

"spark.rapids.sql.test.enabled" -> writeGpu.toString

and

"spark.rapids.sql.test.enabled" -> readGpu.toString

to check all operations running on GPU during GPU read and write separately.

And removed binary test because we don't support binaryType right now.

[res-life]

I also notice that ParquetFilterSuite has testing for the following operators:

      checkFilterPredicate(intAttr === 1, classOf[Eq[_]], resultFun(1))
      checkFilterPredicate(intAttr <=> 1, classOf[Eq[_]], resultFun(1))
      checkFilterPredicate(intAttr =!= 1, classOf[NotEq[_]],
        (2 to 4).map(i => Row.apply(resultFun(i))))

      checkFilterPredicate(intAttr < 2, classOf[Lt[_]], resultFun(1))
      checkFilterPredicate(intAttr > 3, classOf[Gt[_]], resultFun(4))
      checkFilterPredicate(intAttr <= 1, classOf[LtEq[_]], resultFun(1))
      checkFilterPredicate(intAttr >= 4, classOf[GtEq[_]], resultFun(4))

      checkFilterPredicate(Literal(1) === intAttr, classOf[Eq[_]], resultFun(1))
      checkFilterPredicate(Literal(1) <=> intAttr, classOf[Eq[_]], resultFun(1))
      checkFilterPredicate(Literal(2) > intAttr, classOf[Lt[_]], resultFun(1))
      checkFilterPredicate(Literal(3) < intAttr, classOf[Gt[_]], resultFun(4))
      checkFilterPredicate(Literal(1) >= intAttr, classOf[LtEq[_]], resultFun(1))
      checkFilterPredicate(Literal(4) <= intAttr, classOf[GtEq[_]], resultFun(4))

And I notice ParquetFilterSuite also tests V1 and V2 datasources.

Not sure if it's necessary to add above tests.

[res-life]

Can we mark #9119 is invalid?

[thirtiseven]

Thanks for review @res-life

Not sure if it's necessary to add above tests.

Yes there are many cases to test PPD in Spark, I will go through them and add more cases.

Can we mark #9119 is invalid?

Closed it.

[revans2]

nit: This only matters when reading on the CPU. The GPU ignores this.

[revans2]

Actually I think it should be

"spark.rapdis.slq.test.enabled" -> (!writeGpu).toString

or else if we are reading on the GPU, but writing on the CPU we will run into a problem where the we get an error for having things not be on the GPU.

[revans2]

Same here we should update this one similarly to the comment above.

"spark.rapdis.slq.test.enabled" -> (!readGpu).toString,

[res-life]

Added [databricks] in the title to also test databricks.

[revans2]

build

[thirtiseven]

Updated the timestamp test, V2 source also needs spark.sql.parquet.outputTimestampType=TIMESTAMP_MICROS when reading to enable predicate pushdown. Also added date test.

[thirtiseven]

build