-
Notifications
You must be signed in to change notification settings - Fork 7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FPC Codec for floating point data #37553
Conversation
@rschu1ze started reviewing and said that the implementation quality is very good. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi. Thanks for this contribution! I will leave a first batch of comments and hope to continue the review tomorrow (especially the encoding/decoding part). It's mostly minor stuff overall and it should be possible to merge this PR soon-ish.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I reviewed the rest of the change. Overall it looks really good (thanks). My main question at this point would be if the intermediate copy step can somehow be avoided? (see the detailed comments).
|
||
UInt32 CompressionCodecFPC::getMaxCompressedDataSize(UInt32 uncompressed_size) const | ||
{ | ||
auto float_count = (uncompressed_size + float_width - 1) / float_width; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
getMaxCompressedDataSize() does a worst-case estimation of the compressed data size, right?
I don't really understand the calculation in l. 74, i.e. why are we adding float_width - 1? --> Add a comment?
Same for l. 77, why are we adding float_count / 2? A comment for clueless readers like me would be nice ^^
@koloshmet: For documentation, it would be cool if you could also add new items about FPC to [0] and [1]. [0] https://clickhouse.com/docs/en/sql-reference/statements/create/table/#specialized-codecs |
@koloshmet I'll merge this PR now as it is overall of very high quality (thanks again) and the remaining issues are fine to be fixed separately. I can to that as well. Functional tests look good. Performance tests are more interesting:
[0] https://s3.amazonaws.com/clickhouse-test-reports/37553/092a00d95aa1317aa83e7f6e489d1ee00f64d8f5/performance_comparison_aarch64_[1/4]/report.html |
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Implementation of FPC algorithm for floating point data compression
Requested here:
#25925