Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RabbitMQ Memory leak due to reconnection #33698

Closed
Dosexe opened this issue May 30, 2023 · 3 comments
Closed

RabbitMQ Memory leak due to reconnection #33698

Dosexe opened this issue May 30, 2023 · 3 comments
Assignees
Labels

Comments

@Dosexe
Copy link

Dosexe commented May 30, 2023

Describe the bug

If connection is broken and app is reconnected messages are processed as expected, despite one huge thing - under the hood in class io.vertx.rabbitmq.impl.RabbitMQPublisherImpl pendingAcks deque is not cleared anymore, and all messages after reconnection are just stuck there.
pendingAcks.clear() is triggered on reconnection which is correct, but all messages after reconnection are not removed properly due to an Exception in handleConfirmation on confirmation.write().
checkThread() throws an exception during execusion.

Expected behavior

All messages are removed from pendingAcks after reconnection properly. Also it seems that there is no any point of keeping them for an unlimited amount of time there, if messages are not ack in reasonable time there should be an exception for example and messages should be discarded.

Actual behavior

Messages got stuck in RabbitMQPublisherImpl and never removed, which causes a slow memory leak.

How to Reproduce?

Steps to reproduce:

  1. Create an app which uses io.quarkus:quarkus-smallrye-reactive-messaging-rabbitmq
  2. Write code to send messages via @Channel and Emitter (I think the behaviour will be the same with @Outgoing annotation, but didn't test it)
  3. Create a reconnect situation (I just used a debugger to stop processing for some time, on resume connection there were a reconnection with message [io.ver.rab.imp.RabbitMQClientImpl] (vert.x-eventloop-thread-13) reconnect success)

Output of uname -a or ver

Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64 x86_64

Output of java -version

openjdk version "11.0.12" 2021-07-20 OpenJDK Runtime Environment Homebrew (build 11.0.12+0) OpenJDK 64-Bit Server VM Homebrew (build 11.0.12+0, mixed mode)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.16.6.Final;3.0.4.Final

Build tool (ie. output of mvnw --version or gradlew --version)

gradle 7.5.1

Additional information

No response

@cescoffier
Copy link
Member

I cannot say when, but I will have a look.

@cescoffier
Copy link
Member

It's not a reconnection issue (would have been nice, actually), it's a lack of termination.

Will be fixed once smallrye/smallrye-reactive-messaging#2184 is merged, released, and integrated.

@cescoffier
Copy link
Member

Fixed by #34378.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants