fix(rust, python): properly handle json with unclosed strings

[universalmind303]

fixes #5371

A little note about the implementation. The SplitLines iterator is not really designed to handle all the nuances of json parsing. So I decided to go back to leveraging the serde_json:: StreamDeserializer, but now we are using it exclusively as a mechanism to iterate over the rows. We can do this via the RawValue type in serde_json. Which just returns the raw str of each row without doing any parsing of the values, acting as a drop in replacement to SplitLines.

The actual parsing of the rows is still done by simd_json. So the performance impact should be minimal. I'll post some benchmarks once my release build is finished.

[universalmind303]

Suprisingly, this actually gave us some pretty massive performance gains.

%%timeit -n 10
pl.scan_ndjson("big_file.json").collect()

on a 600M file shape: (500001, 18).

release 0.14.25

1.82 s ± 111 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

this branch

236 ms ± 4.99 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

on the "yelp_academic_dataset_review" 5.0G dataset shape: (6990280, 9)

release 0.14.25

error/invalid

this branch

3.72 s ± 881 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

for reference. The scan_csv operation on the same dataset.

2.21 s ± 21.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

[ritchie46]

Thanks!