Add coerce_types flag to parquet ArrowWriter #1938
Labels
enhancement
Any new improvement worthy of a entry in the changelog
good first issue
Good for newcomers
help wanted
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
As discussed in #1666 not all types can be represented within a parquet schema.
Describe the solution you'd like
The consensus appears to be to:
In particular
Date64
If not coerce_types, write as Int64 and embed logical type in arrow schema only. Otherwise case to Date32
Timestamp
If not coerce_types, write as is, setting LogicalType / ConvertedType only where appropriate.
If coerce_types, cast to a UTC timestamp with the closest supported time unit, likely needing #1936.
Interval
If not coerce_types, write as FixedSizeBinaryArray matching the arrow representation and store logical type in arrow schema.
If coerce_types, convert to the relevant parquet representation.
Describe alternatives you've considered
See #1666
The text was updated successfully, but these errors were encountered: