Zstd compression in MN4 fails for large response payloads #11343
Labels
type: bug
Something isn't working
Comments
|
Thanks for the comprehensive report, we will look into it |
|
It falls due to the exception that Netty sends during encoding when it tries to allocate a buffer. The error indicates that bufferSize exceeded the constant variable maxEncodeSize, which is specified in the original netty code by default and cannot be adjusted. if (bufferSize > maxEncodeSize || 0 > bufferSize) {
throw new EncoderException("requested encode buffer size (" + bufferSize + " bytes) exceeds " +
"the maximum allowable size (" + maxEncodeSize + " bytes)");
}switch (encoding) {
case BR -> var10000 = this.makeBrotliEncoder();
case ZSTD -> var10000 = new ZstdEncoder(this.zstdOptions.compressionLevel(), this.zstdOptions.blockSize(), this.zstdOptions.maxEncodeSize());
case SNAPPY -> var10000 = new SnappyFrameEncoder();
case GZIP -> var10000 = ZlibCodecFactory.newZlibEncoder(ZlibWrapper.GZIP, this.gzipOptions.compressionLevel(), this.gzipOptions.windowBits(), this.gzipOptions.memLevel());
case DEFLATE -> var10000 = ZlibCodecFactory.newZlibEncoder(ZlibWrapper.ZLIB, this.deflateOptions.compressionLevel(), this.deflateOptions.windowBits(), this.deflateOptions.memLevel());
default -> throw new IncompatibleClassChangeError();
}final class ZstdConstants {
/**
* Default compression level
*/
static final int DEFAULT_COMPRESSION_LEVEL = Zstd.defaultCompressionLevel();
/**
* Min compression level
*/
static final int MIN_COMPRESSION_LEVEL = Zstd.minCompressionLevel();
/**
* Max compression level
*/
static final int MAX_COMPRESSION_LEVEL = Zstd.maxCompressionLevel();
/**
* Max block size
*/
static final int MAX_BLOCK_SIZE = 1 << (DEFAULT_COMPRESSION_LEVEL + 7) + 0x0F; // 32 M
/**
* Default block size
*/
static final int DEFAULT_BLOCK_SIZE = 1 << 16; // 64 KB
private ZstdConstants() { }
}The only thing that can be done is to output the config parameter to the micronaut to configure maxEncodeSize manually by the user |
|
I assume this is closed via #11361 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected Behavior
A large response payload (>32MB) should be compressed successfully when
zstdcompression is used, just as it is successful when other compression algorithms such asgzipare chosen.Actual Behaviour
When Zstd is used, payloads larger than 32 MB fail to compress. The same payload works fine if gzip is used, eg. the client sends a request header of
Accept-Encoding: gzip, deflate, brbut fails if the header isAccept-Encoding: gzip, deflate, br, zstdThis seems to be caused by the
ZstdOptionsclass passing in a value for maxEncodeSize of 32MB (as defined in theZstdConstantsclass).This has caused some requests for clients to start failing since we upgraded to MN4 (presumably because Zstd wasn't supported in MN and/or Netty prior to this?).
The following error is thrown, and no response is returned to the client at all:
Steps To Reproduce
Clone https://github.com/jonhill1977/MN_zstd_demo (use branch master) then run the app with ./mvnw mn:run
Prove that gzip compression works for both small and large payloads:
curl --location 'http://localhost:8080/small' --header 'Accept-Encoding: gzip, deflate, br' --output -curl --location 'http://localhost:8080/large' \--header 'Accept-Encoding: gzip, deflate, br' --output -Prove that zstd works for small payloads, but fails for large payloads:
curl --location 'http://localhost:8080/small' --header 'Accept-Encoding: gzip, deflate, br, zstd' --output -curl --location 'http://localhost:8080/large' \--header 'Accept-Encoding: gzip, deflate, br, zstd' --output -You will see the first 3 requests all work (albeit curl outputs gibberish since it's compressed), but the 4th request causes an exception on the server. No response is sent to the client at all, and curl just sits waiting for a response.
This is problematic, because requests which used to work prior to MN4 now fail for some users.
The Chrome browser supports Zstd, therefore send an Accept-Encoding header like the second example - this causes Zstd to be used, which fails for large response payloads.
Safari (or a request sent via Postman using default options) does not support Zstd, therefore send an Accept-Encoding header like the first example, and the request works.
Since Zstd support seems to have been added in MN4, this means that users who use Chrome browser now experience failures from endpoints that used to work in previous versions of MN.
It would be good if we could solve this by:
determineEncodingmethod of theio.micronaut.http.server.netty.handler.Compressorclass?). If the payload is too large for a particular compression option, it should choose the "next best" option which can handle the data.Environment Information
Note: I believe both of these are irrelevant to this bug report and the same issue would occur on any OS / JDK.
Example Application
https://github.com/jonhill1977/MN_zstd_demo
Version
4.7.0
The text was updated successfully, but these errors were encountered: