io.netty.util.internal.OutOfDirectMemoryError

[yu199195]

hi, a web service is built with webflux. With time and increased concurrency,
I found that netty's off-heap memory was not freed.(About 3 hours after the service runs)
its, reactor-netty problem?
this is my evn:
spring boot 2.0.3
jdk 1.8
os: liunx
reactor-netty:0.7.7
netty:4.1.17

this is eror:
2018-06-27 09:32:58.217 WARN 6811 --- [server-epoll-11] i.n.c.AbstractChannelHandlerContext : An exception '{}' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 520093703, max: 524288000)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764) ~[netty-buffer-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740) ~[netty-buffer-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-buffer-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:214) ~[netty-buffer-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-buffer-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) ~[netty-buffer-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) ~[netty-buffer-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) ~[netty-buffer-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:137) ~[netty-buffer-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:71) ~[netty-transport-native-epoll-4.1.25.Final-linux-x86_64.jar!/:4.1.25.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:793) ~[netty-transport-native-epoll-4.1.25.Final-linux-x86_64.jar!/:4.1.25.Final]
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:387) [netty-transport-native-epoll-4.1.25.Final-linux-x86_64.jar!/:4.1.25.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-common-4.1.25.Final.jar!/:4.1.25.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:313) [netty-transport-native-epoll-4.1.25.Final-linux-x86_64.jar!/:4.1.25.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.25.Final.jar!/:4.1.25.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]

2018-06-27 09:32:58.217 WARN 6811 --- [server-epoll-16] i.n.c.AbstractChannelHandlerContext : An exception '{}' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:

[violetagg]

@yu199195 run the application with this setting -Dio.netty.leakDetectionLevel=advanced
thus we can detect memory leaks.
https://github.com/netty/netty/wiki/Reference-counted-objects#troubleshooting-buffer-leaks

Also did you set manually the Netty version because Reactor Netty 0.7.7 comes with Netty 4.1.24 as a dependency https://github.com/reactor/reactor-netty/releases/tag/v0.7.7.RELEASE

[yu199195]

i canot find reason。 and alsoi add -Dio.netty.leakDetectionLevel=advanced but not find any log。
jvm not found full gc. everything seems to be all right.
Is the stack caused by excessive concurrency?
spring-boot 2.0.3
reactor-netty 0.7.8
netty 4.1.25

[yu199195]

in my code ,No out-of-heap memory is used . why Out of heap memory overflow? dont understand.
reactor-netty, At high concurrency (20000qps) ,That's a problem?

[violetagg]

Without knowing the scenario it is hard to say whether it is an issue or misconfiguration. Try to increase this -Dio.netty.maxDirectMemory and see what will be the result.

[yu199195]

let me try ，tks

[yu199195]

@violetagg hi，i find netty use DIRECT_MEMORY_COUNTER To count the memory out of the heap
in the PlatformDependent.java 101 line.
i try debug ,It just adds up ,it doesn't decrease.
netty vesion 4.1.25

[smaldini]

If you have a simple reproduction test that would be awesome!

[yu199195]

i use spring-webflux ，In my code, no out-of-heap memory is used，reactor-netty will release buffers？

[violetagg]

@yu199195 were you able to prepare some reproducible scenario?

[violetagg]

Closing this as there is no response from the author. @yu199195 you can reopen it if you have more details about the issue.

[smaldini]

There might be an error (reported elsewhere). I can think of having elastic scheduler at play (or many threads will have an arena cache created). The other thing is webflux/spring webclient might have a niche leak of some sort. It's worth keeping track of it

[smaldini]

Could be related to https://jira.spring.io/browse/SPR-16633

[yu199195]

@smaldini tks

[dave-fl]

Just got it myself.

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 251658247, max: 268435456)

Nothing particularly unique, just Spring Cloud Gateway + some WebClient calls.

[dave-fl]

Spring Webclient just fixed a small leak.

https://jira.spring.io/plugins/servlet/mobile#issue/SPR-17025

[violetagg]

All, try the latest Reactor Netty 0.8.7.BUILD-SNAPSHOT, Reactor Core 3.2.8.RELEASE and Spring Framework 5.1.6.BUILD-SNAPSHOT
Closing this, it there are still memory leaks we will reopen it.

[dave-fl]

Things still leak with the gateway if retry is on and this build.

[violetagg]

@dave-fl Let's track it in the spring gateway issue, if it appear reactor netty we will reopen/create new issue.

[kevinwan1986]

@violetagg
hi,I have the same problem.I can provide more information.
I am using reactor-netty request eth node.

1. First，error is as follows.

2019-07-20 00:10:15.417 ERROR 1 --- [ scheduling-4] c.b.scannerj.quartz.EthQuartzService : scheduled error
com.blockchain.scannerj.exception.ServerSideException: sequenceProcessBlock error blockNum:8184101
at com.blockchain.scannerj.service.impl.EthBlockServiceImpl.sequenceProcessBlock(EthBlockServiceImpl.java:129) ~[classes!/:0.0.1-SNAPSHOT]
at com.blockchain.scannerj.quartz.EthQuartzService.processBlock(EthQuartzService.java:112) ~[classes!/:0.0.1-SNAPSHOT]
at com.blockchain.scannerj.quartz.EthQuartzService.scheduled(EthQuartzService.java:49) ~[classes!/:0.0.1-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) [spring-context-5.1.2.RELEASE.jar!/:5.1.2.RELEASE]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.1.2.RELEASE.jar!/:5.1.2.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.lang.NullPointerException: The mapper returned a null value.
at java.util.Objects.requireNonNull(Objects.java:228) ~[na:1.8.0_181]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:107) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1476) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1476) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:171) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:794) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:560) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.drain(FluxFlatMap.java:540) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.onComplete(FluxFlatMap.java:426) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.DrainUtils.postCompleteDrain(DrainUtils.java:131) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.DrainUtils.postComplete(DrainUtils.java:186) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.FluxMapSignal$FluxMapSignalSubscriber.onComplete(FluxMapSignal.java:213) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:378) ~[reactor-netty-0.8.2.RELEASE.jar!/:0.8.2.RELEASE]
at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:202) ~[reactor-netty-0.8.2.RELEASE.jar!/:0.8.2.RELEASE]
at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:343) ~[reactor-netty-0.8.2.RELEASE.jar!/:0.8.2.RELEASE]
at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:325) ~[reactor-netty-0.8.2.RELEASE.jar!/:0.8.2.RELEASE]
at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:372) ~[reactor-netty-0.8.2.RELEASE.jar!/:0.8.2.RELEASE]
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:522) ~[reactor-netty-0.8.2.RELEASE.jar!/:0.8.2.RELEASE]
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) ~[reactor-netty-0.8.2.RELEASE.jar!/:0.8.2.RELEASE]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[netty-codec-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) ~[netty-codec-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) ~[netty-codec-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) ~[netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar!/:4.1.29.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410) ~[netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar!/:4.1.29.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310) ~[netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar!/:4.1.29.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.29.Final.jar!/:4.1.29.Final]
... 1 common frames omitted
Suppressed: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:93) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at reactor.core.publisher.Mono.block(Mono.java:1475) ~[reactor-core-3.2.2.RELEASE.jar!/:3.2.2.RELEASE]
at com.blockchain.scannerj.service.impl.EthBlockServiceImpl.sequenceProcessBlock(EthBlockServiceImpl.java:114) ~[classes!/:0.0.1-SNAPSHOT]
at com.blockchain.scannerj.quartz.EthQuartzService.processBlock(EthQuartzService.java:112) ~[classes!/:0.0.1-SNAPSHOT]
at com.blockchain.scannerj.quartz.EthQuartzService.scheduled(EthQuartzService.java:49) ~[classes!/:0.0.1-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) [spring-context-5.1.2.RELEASE.jar!/:5.1.2.RELEASE]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.1.2.RELEASE.jar!/:5.1.2.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
... 1 common frames omitted

2.then,after a while

io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source) ~[netty-transport-native-unix-common-4.1.29.Final.jar!/:4.1.29.Final]
2019-07-20 01:00:08.307 ERROR 1 --- [-client-epoll-8] r.n.resources.PooledConnectionProvider : [id: 0xe0d61c54, L:/10.0.0.132:46350 - R:xwallet-eth-main.xender.com/54.169.89.238:80] Pooled connection observed an error

3.At last,as above

' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 3489660935, max: 3502768128)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640) ~[netty-common-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594) ~[netty-common-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764) ~[netty-buffer-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740) ~[netty-buffer-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-buffer-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:214) ~[netty-buffer-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-buffer-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) ~[netty-buffer-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) ~[netty-buffer-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) ~[netty-buffer-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:137) ~[netty-buffer-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:72) ~[netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar!/:4.1.29.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:793) ~[netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar!/:4.1.29.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar!/:4.1.29.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar!/:4.1.29.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar!/:4.1.29.Final]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

[violetagg]

@kevinwan1986 this issue is already closed, please create a new one

[xhAndroid]

netty4默认使用堆外内存，要手动释放。byteBuf.release();

[mariaAbramiuc]

@kevinwan1986 Did you manage to find a fix? I'm also getting this on high load:

2021-04-22 08:31:13.071 WARN 1 --- [tor-tcp-epoll-1] i.n.c.AbstractChannelHandlerContext : An exception 'reactor.core.Exceptions$BubblingException: reactor.netty.ReactorNetty$InternalNettyException: java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:

reactor.core.Exceptions$BubblingException: reactor.netty.ReactorNetty$InternalNettyException: java.lang.OutOfMemoryError: Java heap space
at reactor.core.Exceptions.bubble(Exceptions.java:173) ~[reactor-core-3.3.14.RELEASE.jar!/:3.3.14.RELEASE]
at reactor.core.publisher.Operators.onErrorDropped(Operators.java:635) ~[reactor-core-3.3.14.RELEASE.jar!/:3.3.14.RELEASE]
at io.r2dbc.postgresql.client.ReactorNettyClient$BackendMessageSubscriber.onError(ReactorNettyClient.java:886) ~[r2dbc-postgresql-0.8.7.RELEASE.jar!/:0.8.7.RELEASE]
at reactor.core.publisher.FluxHandle$HandleSubscriber.onError(FluxHandle.java:196) ~[reactor-core-3.3.14.RELEASE.jar!/:3.3.14.RELEASE]
at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onError(FluxPeekFuseable.java:894) ~[reactor-core-3.3.14.RELEASE.jar!/:3.3.14.RELEASE]
at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onError(FluxMap.java:252) ~[reactor-core-3.3.14.RELEASE.jar!/:3.3.14.RELEASE]
at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onError(FluxMap.java:252) ~[reactor-core-3.3.14.RELEASE.jar!/:3.3.14.RELEASE]
at reactor.netty.channel.FluxReceive.onInboundError(FluxReceive.java:415) ~[reactor-netty-0.9.17.RELEASE.jar!/:0.9.17.RELEASE]
at reactor.netty.channel.ChannelOperations.onInboundError(ChannelOperations.java:442) ~[reactor-netty-0.9.17.RELEASE.jar!/:0.9.17.RELEASE]
at reactor.netty.channel.ChannelOperationsHandler.exceptionCaught(ChannelOperationsHandler.java:129) ~[reactor-netty-0.9.17.RELEASE.jar!/:0.9.17.RELEASE]
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:120) ~[reactor-netty-0.9.17.RELEASE.jar!/:0.9.17.RELEASE]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) ~[netty-codec-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[netty-transport-native-epoll-4.1.59.Final-linux-x86_64.jar!/:4.1.59.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-native-epoll-4.1.59.Final-linux-x86_64.jar!/:4.1.59.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-native-epoll-4.1.59.Final-linux-x86_64.jar!/:4.1.59.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.59.Final.jar!/:4.1.59.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.59.Final.jar!/:4.1.59.Final]
at java.base/java.lang.Thread.run(Thread.java:830) ~[na:na]
Caused by: reactor.netty.ReactorNetty$InternalNettyException: java.lang.OutOfMemoryError: Java heap space
Caused by: java.lang.OutOfMemoryError: Java heap space