failed to allocate 16777216 byte(s) of direct memory

[FengXMGeek]

Expected behavior

servers will memory increase smooth where the devices connected to broker dont increase !
where the broker running a week. every client can`t connected to broker . the error message shows "exceptionCaught error is->failed to allocate 16777216 byte(s) of direct memory (used: 2046820359, max: 2051014656)" ?
netstat -nat|grep -i "18883"|wc -l 2638

Actual behavior

Steps to reproduce

Minimal yet complete reproducer code (or URL to code) or complete log file

Moquette MQTT version

Release v.012

JVM version (e.g. java -version)

java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

OS version (e.g. uname -a)

centos 7.4

setup ：
java -server -XX:+UseG1GC -XX:G1RSetUpdatingPauseTimePercent=5 -XX:MaxGCPauseMillis=500 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintPromotionFailure -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Dlog4j.configuration=file:/opt/app/mqtt/moquette-0.20/config/moquette-log.properties -Dmoquette.path=/opt/app/mqtt/moquette-0.20/ -cp /opt/app/mqtt/moquette-0.20/lib/* io.moquette.broker.Server

[djbr-david]

Getting the same problem as above. Stopping the broker seems to be the only way round the issue which isn't ideal. Connecting to the Broker works fine, but publishing a message results in disconnection straight away. Stack trace and DEBUG level log attached from one instance failure. Using version 0.12.1 on Ubuntu 16.04.5 LTS. System works fine for a while - up to about a week - then this starts happening.
mqtt-out-of-direct-memory.txt

[FengXMGeek]

@djbr-david Do you slove this issue ?

[djbr-david]

@FengXMGeek Unfortunately not. Its reproducible though as far as my testing has shown me. Having a broker running with 8 devices connected and sending data on a regular basis - say every 10s or every minute - and each one sends a 20K message if sending every 10s or longer if sending every minute. Using that setup the system seems to be pretty stable. Adding in a single subscriber to take all of the messages eventually makes the system run out of memory. Putting the Broker into Paranoid Leak Detection mode does reveal that messages are leaking. I've also tried a build using the latest Netty (4.1.39) and that still suffers the same issue. Changing QoS to 0 doesn't seem to affect the result either.

[hylkevds]

I've fixed quite a few memory leaks in the last few weeks, and released an unofficial Moquette version to Maven Central in the de.fraunhofer.iosb namespace:

<dependency>
  <groupId>de.fraunhofer.iosb.io.moquette</groupId>
  <artifactId>moquette-broker</artifactId>
  <version>0.14.3</version>
</dependency>

So far it seems quite stable in our tests, with long-lived subscribers that receive several millions of messages a day.