[BUG] String columns written with fastparquet are read differently with Spark RAPIDS

[mythrocks]

Description

As part of #9366, it was noticed that string columns written with fastparquet are read differently on Spark RAPIDS, as compared to vanilla Apache Spark. There seemed to be trailing characters at the end of the last row.

Repro

A gzipped example Parquet file is attached herewith: fastparquet_string.zip.

In the integration tests, the difference manifests as extra characters at the end of the last row. It isn't quite as obvious when read explicitly on Apache Spark and Spark RAPIDS:

// Spark RAPIDS:
scala> spark.read.parquet("/tmp/parq_write").show()
...
+--------------+
|           str|
+--------------+
|           all|
|           the|
|leaves|
+--------------+

// Apache Spark:
scala> spark.read.parquet("/tmp/parq_write").show()
+------+
|   str|
+------+
|   all|
|   the|
|leaves|
+------+

Note that the attachment is a smaller, contrived version of the actual failing test. That file is larger and less readable than the attachment here.

[revans2]

Ya I tested this out myself and it looks like it has to be some kind of a bug in CUDF. CUDF thinks the length of the last string is 14, but spark sees it as 6. Perhaps there is some odd padding that is happening. I am going to try and dump the raw data that we got from CUDF.

[revans2]

So it looks like there are nulls at the end of the last string.

GPU COLUMN LENGTH - NC: 0 DATA: DeviceMemoryBufferView{address=0x30a003400, length=20, id=-1} VAL: DeviceMemoryBufferView{address=0x30a001e00, length=64, id=-1}
COLUMN LENGTH - STRING
0 "all" 616c6c
1 "the" 746865
2 "leaves" 6c65617665730000000000000000

The parquet-mr cli tool appears to dump the data correctly like Spark does.

This even shows up when we try to read str as a binary column instead of a string column. The next step should be to see if we can get a repro case in pure cudf C++ and if so then we need to file an issue with CUDF for this.