You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge?
I'm working on a project that requires using Arrow extension types. Because of the way extension types are encoded in arrow-rs (see apache/arrow-rs#4472) I have to keep track of type information at the field/schema level. This works at execution time, but I've run into an issue trying to extract metadata from a field during the planning phase as field metadata isn't being propagated.
This seems to originate in the ExprSchemable::to_field trait, where when a field is constructed from an expression it only copies the datatype and nullability from the input schema.
Describe the solution you'd like
A reasonable approach to me would be to add a metadata method to the ExprSchemable trait that would forward field metadata for type-preserving expressions (e.g. column, alias, etc.) and return empty metadata for others. ExprSchemable::to_field could then be updated to pass the metadata to the DFField constructor.
Describe alternatives you've considered
The proposed solution would enable metadata propagation through expressions. I haven't gone through every logical plan node so it's possible I'm missing other places where metadata would be erased that would also need to be changed.
Additional context
No response
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem or challenge?
I'm working on a project that requires using Arrow extension types. Because of the way extension types are encoded in
arrow-rs
(see apache/arrow-rs#4472) I have to keep track of type information at the field/schema level. This works at execution time, but I've run into an issue trying to extract metadata from a field during the planning phase as field metadata isn't being propagated.This seems to originate in the
ExprSchemable::to_field
trait, where when a field is constructed from an expression it only copies the datatype and nullability from the input schema.Describe the solution you'd like
A reasonable approach to me would be to add a
metadata
method to theExprSchemable
trait that would forward field metadata for type-preserving expressions (e.g. column, alias, etc.) and return empty metadata for others.ExprSchemable::to_field
could then be updated to pass the metadata to theDFField
constructor.Describe alternatives you've considered
The proposed solution would enable metadata propagation through expressions. I haven't gone through every logical plan node so it's possible I'm missing other places where metadata would be erased that would also need to be changed.
Additional context
No response
The text was updated successfully, but these errors were encountered: