CSVSTAT Memory limits?

[dartdog]

Tried to run csvstat on a 1.9gb file about 7m rows x 74 columns (mixed and sparse) after a long time just got "killed". I'm on a 8GB machine with Linux 14.04 & Python 3+ Is there a way to approximate the limits that can be used? Or can I get a more informative error?

[anthnyprschka]

Same problem here... Tried to csvstat on a ~5gb file on my 4gb machine as i read csvkit computes "line-by-line"?

[jpmckinney]

Not all utilities are line-by-line - calculating statistics requires holding much more of the file in memory.

[anthnyprschka]

Cool thanks for the quick response 👍 found a way to circumvent it

[jpmckinney]

What was your solution?

[anthnyprschka]

To be honest I just switched to using vanilla Python and read the csv line-by-line there. Looping through 2m lines and 9gb took a few seconds for simple stats

[jpmckinney]

Yeah, a generic tool like csvstat does come with significant performance penalties.

[onyxfish]

Closing. csvstat will never be performant for files this large.