Errors in gateway during high load #140
Comments
|
Hmm, seems like a framework error |
|
I see the same behavior too @spencergibb , @fabiormoura - this is for the sample app available here - https://github.com/bijukunjummen/boot2-load-demo |
|
@bijukunjummen @fabiormoura See reactor/reactor-netty@168b2eb. The null pointer exception is fixed by reactor-netty 0.7.3. But that only masks an real problem |
spencergibb
changed the title
spencergibb
changed the title
|
Any ideas on what we could do here? |
|
@scheuchzer is it still a problem? Have you tried with snapshots lately? |
|
@spencergibb Well, the null pointer is gone but I get quite some connection closed exceptions that you mentioned before. Sometimes it's up to 10 percent of the traffic. I'm not sure where to dig. If the target server is really hanging up or if it's somewhere inside netty or reactor. I wrote a simple retry filter that retries on this specific error but this is not a good solution (and the IOException still get's logged somewhere) |
1 similar comment
|
@spencergibb Well, the null pointer is gone but I get quite some connection closed exceptions that you mentioned before. Sometimes it's up to 10 percent of the traffic. I'm not sure where to dig. If the target server is really hanging up or if it's somewhere inside netty or reactor. I wrote a simple retry filter that retries on this specific error but this is not a good solution (and the IOException still get's logged somewhere) |
|
Hi, java.io.IOException: Connection closed prematurely at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:262) at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:110) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420) at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:282) at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1407) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:925) at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) at java.lang.Thread.run(Thread.java:748) |
|
@smaldini or @violetagg any thoughts? |
|
Can you try with Rector Netty 0.7.4.BUILD-SNAPSHOT we have fixes there that might be relevant? Thanks, |
|
After updating the library, the error was gone. Thanks |
|
I can confirm that the errors disappear when using 0.7.4.BUILD-SNAPSHOT |
|
Closing, we can reopen if there is more to discuss. |
|
I still have the issue using 0.7.6. looks like race condition issue with the reactor-netty pool resources. Disabling pool in HttpClient resolved this problem, but that's not ideal. |
|
@anonymousXrandom Hi, Reactor Netty 0.7.7 is the latest release. Can you try that? |
|
@violetagg No issue so far for 0.7.7. It has one line change in isDisposed() function. I think that may be the fix. |
…oud#198) Add support for Composite Discovery client * Test that the default discovery client is present * Return a local Service instance from Composite Discovery Client fixes spring-cloudgh-140
spring-cloud-dependencies: Finchley.M5
spring-boot-starter-parent: 2.0.0.M7
I configured a jmeter test which creates 200 concurrent threads and it hits my gateway. Under this load some request fail with a NullPointerException.
I created two services, the first one using spring cloud gateway and the other one using spring web react. The latter one exposes a /hello GET endpoint and with a delay of 200 ms with the following bean:
The first service has a routing to the second service. After debugging, I noticed what causes the issue is the line below in NettyRoutingFilter, as responseHeaders() returns null:
Full Stacktrace:
And below how the gateway is configured:
The text was updated successfully, but these errors were encountered: