Add support for parsing timestamps from CSV files

[andygrove]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I updated the nyc benchmark schema to use timestamps:

fn nyctaxi_schema() -> Schema {
    Schema::new(vec![
        Field::new("VendorID", DataType::Utf8, true),
        Field::new("pickup_datetime", DataType::Timestamp(TimeUnit::Microsecond, None), true),
        Field::new("dropoff_datetime", DataType::Timestamp(TimeUnit::Microsecond, None), true),
        ...

I tried running a query and got this error.

Error: ArrowError(ExternalError(ArrowError(ParseError("Error while parsing value 2020-01-01 00:35:39 for column 1 at line 2"))))

Describe the solution you'd like
I would like to be able to query CSV files containing timestamps.

Describe alternatives you've considered
None.

Additional context
None.

[alamb]

Arrow contains the code to parse a string --> timestamp correctly here: https://github.com/apache/arrow-rs/blob/master/arrow/src/compute/kernels/cast_utils.rs#L69

This ticket would likely be a matter of hooking that code up into the CSV parser: https://github.com/apache/arrow-rs/blob/master/arrow/src/csv/reader.rs

So most of the code in this PR might best belong in arrow-rs rather than datafusion

[novemberkilo]

I would like to pick this up. Please assign to me as appropriate // @alamb

[novemberkilo]

I have an example of this now on novemberkilo@d9f096a

To reproduce, follow the directions in benchmarks/README.md to get a ballista-scheduler and ballista-executor going locally, then do

cargo run --release --bin nyctaxi -- --iterations 3 --path benchmarks/data/nyctaxi_100.csv --format csv --batch-size 4096

[alamb]

novemberkilo@d9f096a <-- looks very cool 👍

[novemberkilo]

@alamb looks like datafusion is pinned to version 5.3 of arrow-rs. Once apache/arrow-rs#832 is merged, in order to get it to datafusion, will need to upgrade to around 7.0.0 -- that seems like a not-small change? What would the process for this be? Thanks.

[houqp]

@novemberkilo since apache/arrow-rs#832 doesn't break any public api, it will be released as part of arrrow 6.x. @alamb already have a PR ready to merge for arrow-rs 6.x integration: #984. Process wise, we need to get arrow-rs 6.0.0 released first. I will let @alamb decide whether your arrow-rs PR should be merged and released as part of the 6.0.0 release or the release after that.

[alamb]

arrow 6.0.0 is released. When apache/arrow-rs#832 is merged I'll backport that( will be included in 6.1.0, due to be released around Nov 1 2021)

[alamb]

🤔 I wonder if this issue is now done? Or does it need more work?

[novemberkilo]

iirc we just wanted to wait until we could confirm that the version of arrow-rs that contains the fix is being used in datafusion. I don't think it needs more work.