Implement comment.char argument in fread

[arunsrinivasan]

Similar to read.table.

[mattdowle]

Need to ignore whole lines (starting with comment) as well as trailing comments after valid lines.

[mattdowle]

Then update this question : http://stackoverflow.com/questions/18920777/fread-data-table-in-r/

[eddelbuettel]

Bump. Needing this right now.

[DavidArenburg]

Actually fread seem to assume it already has this implemented as it is mentioning comment.char in its warnings. This is a warning I've recently saw (using the dev 1.10.5 version)

Warning in fread(file_x, skip = startind - 1L, header = TRUE, fill = TRUE) :
Stopped early on line 2. Expected 168 fields but found 235. Consider fill=TRUE and comment.char=.

Also, not sure why isn't this an error? It make it harder to catch it with tryCatch

[eddelbuettel]

Interesting. Can you chase the comment and find an author and commit per git blame?
I am mostly using the CRAN version so I work around the issue (when I have to, which is not that often).

[DavidArenburg]

@eddelbuettel You can find it here https://github.com/Rdatatable/data.table/blame/master/src/fread.c#L2351

[eddelbuettel]

Thanks. Which one can click on for git blame so yield ...

Better skip= and nrow= (#2623)

by Matt just one day ago (!!)

[ggirelli]

Bumping too. And if you want to skip only lines with hash at the beginning cmd='grep -v "^#" table.csv'.

[Atrebas]

@ArthurPERE, from ?fread: skip="string" searches for "string" in the file (e.g. a substring of the column names row) and starts on that line, so I think you get the expected result.
You should consider using the following command:

fread("https://github.com/Rdatatable/data.table/files/3162861/Proteome_spodo.as.pfam31.txt", skip = 29, fill = TRUE)

with verbose = TRUE for more details, or the grep method mentioned above.

[jangorecki]

@ArthurPERE it is best to use a documentation as reference and defined behaviour. There you can also find there is no such a thing like comment.char parameter. You can find fread manual at https://rdatatable.gitlab.io/data.table/library/data.table/html/fread.html

AFAIR status of this FR or works on it are well reflected in comments. Be sure to upvote this FR so it will likely speed up its implementation, or at least prioritise. You are also welcome to submit a patch introducing such feature.

[mjsteinbaugh]

Following up on this, I'm open to helping develop and/or test this functionality in a future update

[jangorecki]

@mjsteinbaugh you are very welcome, please submit PR

[dvg-p4]

There is still that error message that references comment.char:

Warning in fread("vep_output") :
  Stopped early on line 4. Expected 6 fields but found 5. Consider fill=TRUE and comment.char=. First discarded non-empty line: <<## Using cache in /home/dgealow/.vep/homo_sapiens/113_GRCh38>>

This is incredibly confusing since comment.char is not a parameter to fread. If there are no plans to actually implement it any time soon (I notice this ticket has been open for 10 years) I'd suggest removing the reference from that error message.

[MichaelChirico]

We have a pending PR: #4486. It requires some love to get it over the line :)