Read action takes too long on a large csv

[briankariuki]

I have a csv of about 25000 records and a resource that uses this csv. Using Api.read takes a very long time even when reading only the first 5 records.

[zachdaniel]

What is a very long time? I'm on a pretty overpowered machine, but a file w/ 20k records just returned for me in .25s. With that said, I'll make some improvements to how our reading logic works to make it leverage the streaming nature of the file read.

[zachdaniel]

After I wrote that I realized that .25s is pretty crazy long time for this to take :) I was just thinking when you said "very long time" that it would be much longer. I'm pushing something to main shortly that improves a fair amount, but likely there are just lots of small optimizations to be made in the way we load the data from the CSV file into their actual resource structs.

[zachdaniel]

Will be looking more into this, as it highlights a few places worth optimizing. I also want to see how much of the slow down is coming from the csv parser vs Ash, but I'd venture a guess that it's mostly loading and validating stored types into memory taking the most time.

[zachdaniel]

I've pushed some non trivial performance improvements up. Give main a try and let me know how it goes :)

[briankariuki]

What is a very long time? I'm on a pretty overpowered machine, but a file w/ 20k records just returned for me in .25s. With that said, I'll make some improvements to how our reading logic works to make it leverage the streaming nature of the file read.

More than a minute I'm afraid. I can share the csv and resource for you to try out.

[briankariuki]

I've pushed some non trivial performance improvements up. Give main a try and let me know how it goes :)

Let me try the main branch and report back.

Also does ash_csv work with ash_json_api? I got some error while trying to read the resource via an api endpoint

[zachdaniel]

No reason it shouldn't. What's the error?