zstd: Another data corruption on SpeedBestCompression level #922
Comments
|
Apparently, the data corruption was introduced between v1.17.2 and v1.17.3. |
|
Thanks! I will investigate ASAP |
|
(got it replicated, but you are catching me at an unfortunate time and I don't have time to work on a fix until Sunday/Monday) |
klauspost
added a commit
that referenced
this issue
Feb 5, 2024
When extending back repeats we may encounter the [`literals_length = 0`](https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#repeat-offsets]) exception, which changes the meaning of offsets. This means the repeat matches the wrong offset. There is little gain extending repeats back anyway, so just avoid it. Fixes #922
|
@MichaelEischer I found the issue and have a fix in #923 This is a recoverable encoding error for once, so if you or anyone else have critical data compressed with this I will be able to detect and correct this issue. If you have such data feel free to attach it here. If there is enough I can build a standalone tool to read and decompress it. |
klauspost
added a commit
that referenced
this issue
Feb 5, 2024
When extending back repeats we may encounter the [`literals_length = 0`](https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#repeat-offsets]) exception, which changes the meaning of offsets. This means the repeat matches the wrong offset. There is little gain extending repeats back anyway, so just avoid it. Fixes #922
|
I will release when I've investigated #920 |
|
New release out. |
|
Thanks for the fix :-) ! |
ItalyPaleAle
added a commit
to ItalyPaleAle/dapr-components-contrib
that referenced
this issue
Feb 24, 2024
Fixes 2 possible data corruption issues. See: - klauspost/compress#875 - klauspost/compress#922 Signed-off-by: ItalyPaleAle <[email protected]>
ItalyPaleAle
added a commit
to ItalyPaleAle/dapr-components-contrib
that referenced
this issue
Feb 24, 2024
Fixes 2 possible data corruption issues. See: - klauspost/compress#875 - klauspost/compress#922 Signed-off-by: ItalyPaleAle <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
After a bug report at restic/restic#4677, we've identified that it is caused by the zstd library at the SpeedBestCompression level. Compressing and decompressing it afterwards results in corrupted data. There is no data corruption at the default compression level. I've tested versions v1.17.4 (via restic using Go 1.21.6) and v1.17.5 (using Go 1.21.6).
minimized.txt (Minimized using the script from #875)
Sorry for only bringing bad news, and that just before the weekend 😞 .
The text was updated successfully, but these errors were encountered: