Switch from CSV to NimbleCSV

[zachdaniel]

The CSV library under the hood uses parallel stream, which for some reason (I suspect this is the source) is emitting a bunch of exit messages. NimbleOptions will perform better anyway I suspect, as we can build the parsers at compile time.

[kevinam99]

Is anyone working on this at the moment? Haven't used NimbleCSV ever but I suppose this would be a good way to start.

[zachdaniel]

Pretty sure no one is, would you like me to assign it to you?

[kevinam99]

Sure. I'd like to have a go at it. Could you give me a brief overview on how this is intended to work?

[zachdaniel]

Honestly, its been a long time since I looked at the code here. Ultimately changing it to use NimbleCSV likely involves something like:

1. a Spark.Dsl.Transformer that creates the compiler for a resource at compile time (i.e does something like this NimbleCSV.define(MyParser, separator: "\t", escape: "\"") (but embedding the resource fields in some way)
2. swap out all of the csv operations to use NimbleCSV via the new parser

[kevinam99]

Okay, cool. Let me take some time to go through it and then maybe I'll create a draft PR.