-
Notifications
You must be signed in to change notification settings - Fork 277
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bottomless: add zstd compression option #468
Conversation
as an env var |
@MarinPostma downgraded to draft, because I see some spontaneous errors when testing xz compression on large random data (one that does not compress well). I'll evalutate other compression implementations as well, because xz support looks a little abandoned in the async_compression crate |
During stress tests, xz turned out to spontaneously fail to compress, same with bzip2. All compression algos are supported by separate crates, so these were simply ruled out. Zstd proved to be: - fast - correct - more than acceptable on compression ratio
Rebranded to zstd. It's a modern one that also powers ScyllaDB, and it first and foremost passed all correctness tests on which xz and bzip2 failed. It's also way faster than any of the other ones, while still providing compression ratios of 12-15 where gzip gives mere 2. |
Gzip does not perform well on data in form of libSQL 4KiB pages, and zstd performed uniformly better in all test cases I covered locally (and not worse in case of random data with super high entropy).
Transplanted from libsql/sqld#780