Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure buffers are flushed to avoid decompression stream underread #251

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Commits on Oct 19, 2023

  1. Add a test demonstrating underread of underlying buffers

    When reading from a decompressor that was constructed using `Decoder::with_buffer`, the decoder may not consume all of the input by the time it returns all of the decompressed output.  This means when you call `.finish()` on the decoder to get the underlying stream back, it is not pointing after the end of the compressed data.
    
    This commit adds a test that demonstrates the issue.
    markbt committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    5161ab0 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2023

  1. Ensure reader buffer is flushed when extracting reader

    After the decoder stream has yielded all of the uncompressed data, it is possible for the input stream to still not be fully consumed.  This means if we extract the inner stream at this point, it will not be pointing to the end of the compressed data.
    
    From the [zstd documentation](https://facebook.github.io/zstd/zstd_manual.html#Chapter9) for `ZSTD_decompressStream`:
    
    > But if `output.pos == output.size`, there might be some data left within internal buffers.
    > In which case, call ZSTD_decompressStream() again to flush whatever remains in the buffer.
    
    This is only necessary if the caller wants the stream back, so at that point we can force an additional call to `ZSTD_decompressStream` by reading to a zero-length buffer.
    markbt committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    ea02a36 View commit details
    Browse the repository at this point in the history