[FEA] Audit_3.0.1: Test for not ignoring rows in CSV that start with null char. Test in cudf

[razajafri]

commit 68ff809060c312fee1fbe8d46657e6c7aec62dba
Author: Sean Owen [email protected]
Date: Wed Aug 26 00:25:58 2020 +0900

[SPARK-32614][SQL] Don't apply comment processing if 'comment' unset for CSV

[kuhushukla]

@razajafri why do we need to audit this any further? COuld you elaborate here? Thanks!

[razajafri]

To support this we will need to make change in cudf to expose an API for the user to set/unset whether to ignore the lines starting with commentPrefix.

[revans2]

@razajafri if I read through the cudf code '\0' appears to disable comments

https://github.com/rapidsai/cudf/blob/aaba25034c3b927fc1fb7a80ab4947eb06a6c6b5/cpp/src/io/csv/csv_gpu.cu#L1015
https://github.com/rapidsai/cudf/blob/aaba25034c3b927fc1fb7a80ab4947eb06a6c6b5/cpp/src/io/csv/csv_gpu.cu#L1115

So what we would need to do is to fall back to the CPU if someone actually set the comment character to '\0' instead of going off of the default. We probably also should update the docs in cudf to explain this.

Because this is CSV and only for a very small corner case I don't think it is that critical to fix right now so dropping the priority.