Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Reading Alternative List Representations to Arrow From Parquet #1680

Closed
tustvold opened this issue May 9, 2022 · 0 comments · Fixed by #1682
Closed

Support Reading Alternative List Representations to Arrow From Parquet #1680

tustvold opened this issue May 9, 2022 · 0 comments · Fixed by #1682
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate

Comments

@tustvold
Copy link
Contributor

tustvold commented May 9, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently the parquet -> arrow conversion supports the 3-level representation of lists, as described here.

However, there is limited support for alternative representations as described here. In particular

  • No test coverage of fallback representations for List annotated types
  • No support for repeated fields outside of List annotated types

Describe the solution you'd like

We should support the parquet specification
@tustvold tustvold added the enhancement Any new improvement worthy of a entry in the changelog label May 9, 2022
@tustvold tustvold changed the title Support Reading Alternative List Representations to Parquet Support Reading Alternative List Representations to Arrow From Parquet May 9, 2022
@tustvold tustvold mentioned this issue May 9, 2022
Closed
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 9, 2022
@tustvold
Separate parquet -> arrow conversion logic (apache#1655)
4b98333 
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)

Fix schema inference for repeated fields (apache#1681)

Support reading alternative list representations from parquet (apache#1680)
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 9, 2022
@tustvold
Separate parquet -> arrow conversion logic (apache#1655)
6eb932c 
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)

Fix schema inference for repeated fields (apache#1681)

Support reading alternative list representations from parquet (apache#1680)

Consistent handling of unsupported arrow types in parquet (apache#1666)
@tustvold tustvold mentioned this issue May 9, 2022
Merged
tustvold added a commit to tustvold/arrow-rs that referenced this issue May 11, 2022
@tustvold
Separate parquet -> arrow conversion logic (apache#1655)
e2f12de 
Don't treat embedded arrow schema as authoritative (apache#1663)

Fix projection of nested parquet files (apache#1652) (apache#1654)

Fix schema inference for repeated fields (apache#1681)

Support reading alternative list representations from parquet (apache#1680)
alamb pushed a commit that referenced this issue May 13, 2022
@tustvold
Fix Parquet Arrow Schema Inference (#1682)
5b154ea 
* Separate parquet -> arrow conversion logic (#1655)

Don't treat embedded arrow schema as authoritative (#1663)

Fix projection of nested parquet files (#1652) (#1654)

Fix schema inference for repeated fields (#1681)

Support reading alternative list representations from parquet (#1680)

* Add more tests

* Pass pointers by reference

* More docs

* Fix lint

* Review feedback

* Review feedback

* Fix test failures related to #1697
@alamb alamb added the parquet Changes to the parquet crate label May 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Parquet Reader's Arrow Schema Inference tustvold/arrow-rs
2 participants
@alamb @tustvold