Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: prevent empty stats struct during parquet write #2125

Merged
merged 5 commits into from
Jan 28, 2024

Conversation

alexwilcoxson-rel
Copy link
Contributor

Description

When building the arrow schema for delta checkpoints, List, Map, and Binary max/min stats are not collected. If you have a Struct column with only a List Map, or Binary field, then the arrow schema gets an empty Struct. Parquet writer fails with this:

ParquetParseError { source: ArrowError("Parquet does not support writing empty structs") }

@rtyler rtyler enabled auto-merge (squash) January 27, 2024 00:35
@rtyler rtyler merged commit 0f6790f into delta-io:main Jan 28, 2024
20 checks passed
RobinLin666 pushed a commit to RobinLin666/delta-rs that referenced this pull request Feb 2, 2024
# Description
When building the arrow schema for delta checkpoints, List, Map, and
Binary max/min stats are not collected. If you have a Struct column with
only a List Map, or Binary field, then the arrow schema gets an empty
Struct. Parquet writer fails with this:

```
ParquetParseError { source: ArrowError("Parquet does not support writing empty structs") }
```
rtyler added a commit that referenced this pull request Feb 21, 2024
@alexwilcoxson-rel alexwilcoxson-rel deleted the checkpoint-empty-struct branch May 9, 2024 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/rust Issues for the Rust crate crate/core
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants