Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Impl RleV1Encoder for integer #37

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

suxiaogang223
Copy link

Copy link
Collaborator

@Jefffrey Jefffrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising this, have left some comments 🙂

src/encoding/integer/mod.rs Outdated Show resolved Hide resolved
src/encoding/integer/rle_v1.rs Show resolved Hide resolved
@Jefffrey
Copy link
Collaborator

Jefffrey commented Jan 4, 2025

Looking better now. Some CI failures to address (can ignore the Cargo Deny License Check failure)

@suxiaogang223
Copy link
Author

suxiaogang223 commented Jan 5, 2025

@Jefffrey I use a fixed-size array on the stack instead of a heap-allocated Vec now. Perhaps we can use the same approach in the implementation of rle_v2 to avoid memory allocation. https://github.com/datafusion-contrib/orc-rust/blob/main/src/encoding/integer/rle_v2/mod.rs#L217

Copy link
Collaborator

@Jefffrey Jefffrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like where this is headed 👍

src/encoding/integer/rle_v1.rs Outdated Show resolved Hide resolved
src/encoding/integer/rle_v1.rs Outdated Show resolved Hide resolved
src/encoding/integer/rle_v1.rs Outdated Show resolved Hide resolved
src/encoding/integer/rle_v1.rs Outdated Show resolved Hide resolved
src/encoding/integer/rle_v1.rs Outdated Show resolved Hide resolved
src/encoding/integer/rle_v1.rs Outdated Show resolved Hide resolved
@suxiaogang223
Copy link
Author

I think it is feasible to use Rc<RefCell> to reuse buffers on the heap. You can check the changes of the last commit.

@Jefffrey
Copy link
Collaborator

I think it is feasible to use Rc<RefCell> to reuse buffers on the heap. You can check the changes of the last commit.

At this point it might just be easier to have the Vec directly in the RleV1Encoder itself and not bother having it also in RleV1EncodingState::Literal; I feel that is simpler than having to wrap it in an Rc Refcell

@suxiaogang223
Copy link
Author

I think it is feasible to use Rc<RefCell> to reuse buffers on the heap. You can check the changes of the last commit.

At this point it might just be easier to have the Vec directly in the RleV1Encoder itself and not bother having it also in RleV1EncodingState::Literal; I feel that is simpler than having to wrap it in an Rc Refcell

Yes, I think I'm overcomplicating things...

@suxiaogang223
Copy link
Author

@Jefffrey Done, can you please review this pr again?🤓

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Integer RLE encoding
2 participants