Support documents larger than 512MB by streaming the contents

[js-9]

This is a common issue and can be fixed using a variety of methods, but I'm too lazy and too frustrated right now to do a pr. One example would be to use big-json.

[josdejong]

Thanks for your input @js-9

What is "large" to you? What kind of document sizes are you talking about?

Interesting idea to make at least the CLI stream based like big-json, that should be possible I think. The parser itself can use some optimization too, it could be faster, but even when not the fastest it would be nice to be able to repair documents in a streaming way.

[js-9]

I was working on a pretty big geojson dataset of 783MB. The limitation of modern nodejs, as I've found out, for string size is 512MB.

[josdejong]

👍 ah, yes 512 MB is the limit.

[josdejong]

According to MDN, we should be able to handle up to 1 GB in node.js. When testing this out files that large though, the library runs out of memory. I'm not sure if we can improve the memory usage of the library itself (unless we build it in a streaming way).

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/length

[josdejong]

To solve this we should implement a streaming parser. I don't expect to implement this any time soon, therefore I'll move this idea from the issues section to an idea in Discussions.

[josdejong]

FYI: I was experimenting last Friday to see what's involved to support streaming JSON. It's not trivial, but I'm working on it.

[josdejong]

Streaming support is now available in [email protected] 🎉