Faster import 2

[laem]

Closes #170 to drop the first technique of using the CLI sqlite3 .import option, as it results in check errors and the IO to write the CSV is too slow.

In this PR, we're keeping the in-js sqlite3 import but we're keeping other optimizations developed in the first PR.

[laem]

Step result : for a rather large dataset (Bretagne), we're getting 40 seconds of processing compared to the 1minute 04 seconds of the master branch.

The last optimisation attempt, db.transaction() does not appear to speed things up, but is cleaner IMHO.

Note : some tests are failing. One about shapes (I didn't look into it) and one more worrysome about get-stops.

[laem]

@brendannee I can't find why this test fails : getShapesAsGeoJSON(): › should return geojson with shapes if they exist.

Could you look into it ?

[laem]

Interesting ! I get a heap limit error when trying the big paris region GTFS (~1go).
I need to see if's it doesn't break the concept of transaction prepare of my pr.

OK, I see what's problematic : in master, at each batch, lines are popped from the main lines object. The whole stoptimes just does not fit in the heap size anymore because of list[].

Running with --max-old-space-size=16384 solves the problem, but of course I guess you don't want to impose this limitation. The size of the array could be limited like it was before but with a value higher than the old maxInsertVariables = 32_000;.

For the Parisian region, it lowers the import time from 4:34 to 2:50 :)

[laem]

Result for my whole GTFS collection so far (one part of France, let's say 1/3rd).

[this branch] Parse GTFS: 10:26.820 (m:ss.mmm)
[master] Parse GTFS: 16:44.851 (m:ss.mmm)

Final db size : 9,4 Go / 11Go.

[laem]

Wait @brendannee some tests did not pass 😅

[brendannee]

Thanks so much for this big improvement.

I released a new version: https://github.com/BlinkTagInc/node-gtfs/releases/tag/4.15.0 (with a few other reorganizations).

Let me know what you think and if you have other ideas for improvements.

[brendannee]

I fixed the issue with the tests that did not pass - so it should be good to go.