zstd: Tweak DecodeAll standard allocs #295
Conversation
Default to compressed size * 2 if no frame size and no destination buffer given.
| size := frame.WindowSize * 2 | ||
| // Allocate len(input) * 2 by default if nothing is provided | ||
| // and we didn't get frame content size. | ||
| size := len(input) * 2 |
There was a problem hiding this comment.
With compression ratio over 2, does this lead to the sequence decoder reallocating to 2MB maxBlockSize? Could that mean that with this change the library allocates 2MB rather than 1MB (previous cap) for small frames?
There was a problem hiding this comment.
@larry-cdn77 The 1MB cap check is still, see next lines.
There was a problem hiding this comment.
@larry-cdn77 And you can always just allocate with any custom size and send that as the destination.
There was a problem hiding this comment.
Certainly. Thank you for responding.
I wonder if I am right to observe that in certain situations one has to hit a sweetspot in the size of custom slice to pass in. I have a high throughput application with message sizes of less than 1KB. The custom slice passed in has to be relatively small to avoid garbage collection spinning up to a lot of CPU time (I've seen a 50% throughput penalty). And it has to big enough to stay clear of sequence decoder replacing it with a new 2MB one, again degrading performance.
Currently the size of my slice is a multiple of compressed size, and experimentally a multiplier of 32 appears to be the sweetspot (I didn't fully understand if this is how much compression my encoder has achieved, or there are other factors behind that number).
Related to this, would it make sense to adjust the library's maxBlockSize given that my data is zstd library encoded and they seem to use 128KB maximum block size?
There was a problem hiding this comment.
@larry-cdn77 If you are using this package to compress it doesn't matter, since it will have FrameContentSize which allows to make an exact allocation and this will never be relevant.
a multiplier of 32
So you have a 32:1 compression ratio? That is way above what most will see, so I can't use that metric. With an expected factor of 2 we should be able to decode most content with 1 or 2 allocations.
maxBlockSize is defined by the format, so it cannot be changed. This is the maximum size of an uncompressed block.
We could take a look at #253 which could be made to decode the frame and first block headers and return the size if there is only one block.
Default to compressed size * 2 if no frame size and no destination buffer given.