Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Go][Parquet] Handle Boolean RLE encoding/decoding #38462

Closed
zeroshade opened this issue Oct 25, 2023 · 0 comments · Fixed by #38367
Closed

[Go][Parquet] Handle Boolean RLE encoding/decoding #38462

zeroshade opened this issue Oct 25, 2023 · 0 comments · Fixed by #38367
Assignees
@zeroshade
Labels
Component: Go Component: Parquet Type: enhancement
Milestone
15.0.0

Comments

@zeroshade
Copy link
Member

Describe the enhancement requested

In addition to the plain boolean encoding (done as a bitmap) we should also support the bit-packed RLE/hybrid boolean encoding/decoding for parquet files in the Go Parquet implementation. This is also relevant more now because several of the parquet-testing repo files have been updated to utilize boolean columns that have the RLE encoding type.

Component(s)

Go, Parquet
@github-actions github-actions bot mentioned this issue Oct 25, 2023
Merged
zeroshade added a commit that referenced this issue Oct 30, 2023
@zeroshade @kou
GH-38462: [Go][Parquet] Handle Boolean RLE encoding/decoding (#38367)
23b62a4 
### Rationale for this change
Looks like the parquet-testing repo files have been updated and now include boolean columns which use the RLE encoding type. This causes the Go parquet lib to fail verification tests when it pulls the most recent commits for the parquet-testing repository. So a solution for this is to actually implement the RleBoolean encoder and decoder.

### What changes are included in this PR?
Adding `RleBooleanEncoder` and `RleBooleanDecoder` and updating the `parquet-testing` repo.

### Are these changes tested?
Unit tests are added, and this is also tested via the `parquet-testing` golden files.

* Closes: #38345
* Closes: #38462

Lead-authored-by: Matt Topol <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Matt Topol <[email protected]>
@zeroshade zeroshade added this to the 15.0.0 milestone Oct 30, 2023
loicalleyne pushed a commit to loicalleyne/arrow that referenced this issue Nov 13, 2023
@zeroshade @kou @loicalleyne
apacheGH-38462: [Go][Parquet] Handle Boolean RLE encoding/decoding (a…
6dc72cd 
…pache#38367)

### Rationale for this change
Looks like the parquet-testing repo files have been updated and now include boolean columns which use the RLE encoding type. This causes the Go parquet lib to fail verification tests when it pulls the most recent commits for the parquet-testing repository. So a solution for this is to actually implement the RleBoolean encoder and decoder.

### What changes are included in this PR?
Adding `RleBooleanEncoder` and `RleBooleanDecoder` and updating the `parquet-testing` repo.

### Are these changes tested?
Unit tests are added, and this is also tested via the `parquet-testing` golden files.

* Closes: apache#38345
* Closes: apache#38462

Lead-authored-by: Matt Topol <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Matt Topol <[email protected]>
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
@zeroshade @kou @dgreiss
apacheGH-38462: [Go][Parquet] Handle Boolean RLE encoding/decoding (a…
cea8532 
…pache#38367)

### Rationale for this change
Looks like the parquet-testing repo files have been updated and now include boolean columns which use the RLE encoding type. This causes the Go parquet lib to fail verification tests when it pulls the most recent commits for the parquet-testing repository. So a solution for this is to actually implement the RleBoolean encoder and decoder.

### What changes are included in this PR?
Adding `RleBooleanEncoder` and `RleBooleanDecoder` and updating the `parquet-testing` repo.

### Are these changes tested?
Unit tests are added, and this is also tested via the `parquet-testing` golden files.

* Closes: apache#38345
* Closes: apache#38462

Lead-authored-by: Matt Topol <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Matt Topol <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zeroshade zeroshade

Labels
Component: Go Component: Parquet Type: enhancement
Projects
None yet
Milestone
15.0.0
Development

Successfully merging a pull request may close this issue.

GH-38462: [Go][Parquet] Handle Boolean RLE encoding/decoding zeroshade/arrow
1 participant
@zeroshade