Feat/replace csv parser by papa parse #14
Conversation
|
Processing the CSV as is currently done is taking considerably more time than with the old method. I'm going to see what I can do to improve the efficiency of the parsing. I would wait to review this until I have found a better solution. |
lvancraen
changed the title
lvancraen
changed the title
|
I have updated the code to run MUCH MUCH faster. Instead of processing each row individually within Papa parse, I parse the entire CSV then process each row. This increased the speed by a factor of 10. I tested on WINDON, and previously it was taking ~550secs to run CLE. Now, it only takes ~45secs. You can now review 🙏 |
| } | ||
|
|
||
| row = row.trim(); | ||
| const trimmedRow = row.map(value => value.trim()); | ||
| const gtfsRow = new GtfsRow(trimmedRow); |
There was a problem hiding this comment.
You're recreating the GtfsRow for each row, which defeats the memory saving purpose (creating the class n time + object n time, instead of class 1 time + object n time). The keys will be copied the same way than before the optimization.
There was a problem hiding this comment.
Great point. I will update this
Feature
gtfsNodeLib nows uses Papa Parse to parse through the CSV instead of the internal CSV parser.