Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C++][Python] Support reading/writing key-value metadata from/to ColumnChunkMetaData #41579

Closed
clee704 opened this issue May 8, 2024 · 1 comment
Closed

[C++][Python] Support reading/writing key-value metadata from/to ColumnChunkMetaData #41579

clee704 opened this issue May 8, 2024 · 1 comment
Assignees
@clee704
Labels
Component: C++ Type: enhancement
Milestone
18.0.0

Comments

@clee704
Copy link
Contributor

clee704 commented May 8, 2024

Describe the enhancement requested

According to the standard, ColumnMetaData can have key_value_metadata which is a list of KeyValue.

https://parquet.apache.org/docs/file-format/metadata/

image

Component(s)

C++
@github-actions github-actions bot mentioned this issue May 8, 2024
Merged
@clee704 clee704 changed the title Support reading/writing key-value metadata from/to ColumnMetaData [C++][Python] Support reading/writing key-value metadata from/to ColumnMetaData May 8, 2024
@clee704 clee704 changed the title [C++][Python] Support reading/writing key-value metadata from/to ColumnMetaData [C++][Python] Support reading/writing key-value metadata from/to ColumnChunkMetaData May 8, 2024
mapleFU added a commit that referenced this issue Aug 15, 2024
@clee704 @mapleFU
GH-41579: [C++][Python][Parquet] Support reading/writing key-value me…
2767dc5 
…tadata from/to ColumnChunkMetaData (#41580)

### Rationale for this change
Parquet standard allows reading/writing key-value metadata from/to ColumnChunkMetaData, but there is no way to do that with Parquet C++.

### What changes are included in this PR?
Support reading/writing key-value metadata from/to ColumnChunkMetaData with Parquet C++ reader/writer. Support reading key-value metadata from ColumnChunkMetaData with pyarrow.parquet.

### Are these changes tested?
Yes, unit tests are added

### Are there any user-facing changes?
Yes.
- Users can read or write key-value metadata for column chunks with Parquet C++.
- Users can read key-value metadata for column chunks with PyArrow.
- parquet-reader tool prints key-value metadata in column chunks when `--print-key-value-metadata` option is used.

* GitHub Issue: #41579

Lead-authored-by: Chungmin Lee <[email protected]>
Co-authored-by: mwish <[email protected]>
Signed-off-by: mwish <[email protected]>
@mapleFU mapleFU added this to the 18.0.0 milestone Aug 15, 2024
@mapleFU
Copy link
Member

mapleFU commented Aug 15, 2024

Issue resolved by pull request 41580
#41580

@mapleFU mapleFU closed this as completed Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@clee704 clee704

Labels
Component: C++ Type: enhancement
Projects
None yet
Milestone
18.0.0
Development

No branches or pull requests
2 participants
@clee704 @mapleFU