Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preserve field metadata across expressions in logical plans #6886

Closed
sydduckworth opened this issue Jul 7, 2023 · 0 comments · Fixed by #6920
Closed

Preserve field metadata across expressions in logical plans #6886

sydduckworth opened this issue Jul 7, 2023 · 0 comments · Fixed by #6920
Labels
enhancement New feature or request

Comments

@sydduckworth
Copy link
Contributor

Is your feature request related to a problem or challenge?

I'm working on a project that requires using Arrow extension types. Because of the way extension types are encoded in arrow-rs (see apache/arrow-rs#4472) I have to keep track of type information at the field/schema level. This works at execution time, but I've run into an issue trying to extract metadata from a field during the planning phase as field metadata isn't being propagated.

This seems to originate in the ExprSchemable::to_field trait, where when a field is constructed from an expression it only copies the datatype and nullability from the input schema.

Describe the solution you'd like

A reasonable approach to me would be to add a metadata method to the ExprSchemable trait that would forward field metadata for type-preserving expressions (e.g. column, alias, etc.) and return empty metadata for others.
ExprSchemable::to_field could then be updated to pass the metadata to the DFField constructor.

Describe alternatives you've considered

The proposed solution would enable metadata propagation through expressions. I haven't gone through every logical plan node so it's possible I'm missing other places where metadata would be erased that would also need to be changed.

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant