[BUG] CSV reading null inconsistent between spark.rapids.sql.format.csv.enabled=true&false

[mattf]

Describe the bug

input...

empty-0space,,end
empty-0space-quoted,"",end
empty-1space, ,end
empty-1space-quoted," ",end
empty-2space,  ,end
empty-2space-quoted,"  ",end
no-null,3.14,end

schema...

StructType([StructField("firstField", StringType()),
            StructField("dblField", DoubleType(), True),
            StructField("lastField", StringType())])

config spark.rapids.sql.format.csv.enabled = true ...

+-------------------+--------+---------+
|         firstField|dblField|lastField|
+-------------------+--------+---------+
|       empty-0space|    null|      end|
|empty-0space-quoted|    null|      end|
|       empty-1space|     0.0|      end|
|empty-1space-quoted|    null|      end|
|       empty-2space|     0.0|      end|
|empty-2space-quoted|    null|      end|
|            no-null|    3.14|      end|
+-------------------+--------+---------+

config spark.rapids.sql.format.csv.enabled = false ...

+-------------------+--------+---------+
|         firstField|dblField|lastField|
+-------------------+--------+---------+
|       empty-0space|    null|      end|
|empty-0space-quoted|    null|      end|
|       empty-1space|    null|      end|
|empty-1space-quoted|    null|      end|
|       empty-2space|    null|      end|
|empty-2space-quoted|    null|      end|
|            no-null|    3.14|      end|
+-------------------+--------+---------+

Expected behavior
results should be the same

Environment details (please complete the following information)

• Environment location: local
• Spark configuration settings related to the issue: 0.4 w/ libcudf 0.18.1

[mattf]

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, DoubleType

INPUT = \
"""
empty-0space,,end
empty-0space-quoted,"",end
empty-1space, ,end
empty-1space-quoted," ",end
empty-2space,  ,end
empty-2space-quoted,"  ",end
no-null,3.14,end
"""

with open("repro-data.csv", "w") as fp:
    fp.write(INPUT)

schema = StructType([StructField("firstField", StringType()), StructField("dblField", DoubleType(), True), StructField("lastField", StringType())])

spark = SparkSession.builder.config("spark.plugins", "com.nvidia.spark.SQLPlugin").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")

print("input...")
print(INPUT)
print("schema:", schema)

print("spark.rapids.sql.format.csv.enabled =", spark.conf.get("spark.rapids.sql.format.csv.enabled"))
spark.read.csv("repro-data.csv", schema=schema).show()

[revans2]

So just to clarify a bit. By default Spark treats an empty string as null. When Spark parses " " or " " it sees it as a string value with one or two spaces in it. Then it tries to parse that string as a floating point value. That fails because they do not match the correct pattern for a float so Spark inserts a null. cudf is not doing this because... Well we turned on CSV parsing knowing that there are a lot of incompatibility in it, but because it is so fast on the GPU vs the CPU we wanted to do it anyways. I think in the short term we are going to disable CSV parsing by default, and longer term look at trying to enable it selectively for types where we know that we can either have cudf parse it correctly or we can work around it after the fact.

[sameerz]

Adding to 0.5 to set spark.rapids.sql.format.csv.enabled to false by default.

[sameerz]

The PR to disable csv by default is #2072 . There are other longer term fixes required here, so leaving this open.