unable to read delta table when table contains both null and non-null add stats #2477
Labels
bug
Something isn't working
Comments
ion-elgreco
pushed a commit
that referenced
this issue
May 6, 2024
…nd non-null add stats (#2476) # Description To fix the issue when a delta table contains add action with stats_parsed: null. As shown in the test case, `001.json` contains an Add action with stats, while `002.json` contains an Add action with `stats_parsed: null`, before this fix, it will complain: ``` Arrow { source: InvalidArgumentError("all columns in a record batch must have the same length") } ``` The issue is that the array for `num_records` has two values, while for other stats such as null_count, the None value is filtered out by `flat_map`, so there is only one value in the array. # Related Issue(s) closes #2477 # Documentation <!--- Share links to useful documentation --->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Environment
Delta-rs version: the latest
Binding: rust
Environment:
Bug
What happened:
Unable to read table when it contains both commit of add action with stats, and commit of add action without stats.
What you expected to happen:
Able to read the file whether stats exists for all actions
How to reproduce it:
A simple table with several add actions, either in one commit or several commits. Some of the add action has stats, while some has
stats_parsed: null.More details:
The text was updated successfully, but these errors were encountered: