mlr silently truncating some TSV files since #923

[johnkerl]

Since #923 , mlr is silently truncating some files. I do not know which files, and cannot share the one file I know is causing the issue because of data security issues. Here's the output:

original file

$ wc -l temp.tsv
308170 temp.tsv

tsv filetype

$ mlr --tsv cat temp.tsv | wc -l
20408

csv filetype with tab separator

$ mlr --csv --fs "\t" cat temp.tsv | wc -l
308170

version 6.0.0

$ mlr --tsv cat temp.tsv | wc -l
308170

Even if my file is somehow invalid, this probably should throw an error instead of silently truncating the file. Is there something I should be looking for in the file so I can make a reprex for you?

Originally posted by @BEFH in #923 (comment)

[BEFH]

PR is #923, thanks.

[johnkerl]

This is fixed for TSV (in my code) but the issue persists for CSV where the bug is in the Go-CSV library -- this needs to be worked around as well ...

[BEFH]

Very interesting. For me, using `--csv --fs "\t"` worked previously. I'll test the fix and get back to you on if it works.

…

On Thu, Dec 29, 2022 at 2:34 PM John Kerl ***@***.***> wrote: This is fixed for TSV (in my code) but the issue persists for CSV where the bug is in the Go-CSV library -- this needs to be worked around as well ... — Reply to this email directly, view it on GitHub <#1102 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZ2Z2BXEON3PODBT47SVNLWPXRV5ANCNFSM6AAAAAAQ64GZRY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[johnkerl]

Concretely:

$ cat test.csv
a
1
2

4
5

$ mlr5 --icsv --ojson cat test.csv
{ "a": 1 }
{ "a": 2 }
{ "a": "" }
{ "a": 4 }
{ "a": 5 }

$ mlr --icsv --ojson cat test.csv
[
{
  "a": 1
},
{
  "a": 2
},
{
  "a": 4
},
{
  "a": 5
}
]

[johnkerl]

This is fixed for TSV (in my code) but the issue persists for CSV where the bug is in the Go-CSV library -- this needs to be worked around as well ...

@skitt has kindly opened #1164 to track the problem for CSV so I'll close this issue out as it solves the problem for TSV.