Memory leak in Tomcat

[eduponte]

After stopping, undeploying or redeploying a Hystrix-powered webapp in Tomcat we find some errors as the following

SEVERE: The web application [/api-0.0.1-SNAPSHOT] appears to have started a thread named [hystrix-ProductsGroup-1] but has failed to stop it. This is very likely to create a memory leak.

Additionally, using JConsole we can see the threads, i.e. hystrix-ProductsGroup-1, are still alive.

We have tried to take control of the thread destruction setting withKeepAliveTimeMinutes as described here https://github.com/Netflix/Hystrix/wiki/Configuration , but found that the parameter is no longer available (1.2.5).

As a result, after a number of deploys, Tomcat ends with a PermGen error.

Can Hystrix remove these threads? Is there an alternative way we can get rid of the old threads?

Thanks in advance.

[daveray]

Hi. You should be able to use com.netflix.hystrix.Hystrix.reset() to clean up the thread pools and other resources created by Hystrix. It was added in commit 976f2fb and should be available in version 1.2.8. Cheers.

[eduponte]

@daveray Thanks very much.

After upgrading to 1.2.8 and adding the reset in a Listener, all the hystrix-Whatever-i threads are gone.

That's pretty much what we wanted, although a new thread HystrixTimer_Tick appears and is not removed after undeploy:

SEVERE: The web application [/api-0.0.1-SNAPSHOT] appears to have started a thread named [HystrixTimer_Tick] but has failed to stop it. This is very likely to create a memory leak.

Thread details form JConsole:

Name: HystrixTimer_Tick
State: TIMED_WAITING
Total blocked: 0 Total waited: 29.389

Stack trace:
java.lang.Thread.sleep(Native Method)
com.netflix.hystrix.util.HystrixTimer$TickThread.run(HystrixTimer.java:171)

[daveray]

Yep. I guess the timer thread was missed. I'll leave it up to Ben whether we track that here, or reopen #102 .

[benjchristensen]

I'll get to this ... I've been swamped the last couple days but it's near the top of my list.

[eduponte]

We will keep an eye on it. Thanks!

[benjchristensen]

Can someone please review pull request #111 to ensure the concurrency code is correct? Unit tests all suggest it's good ... but it's concurrency so another set of eyes would be good.

The thread shutdown should work with a thread interrupt or Hystrix.reset (that just uses interrupt to shut it down). The important part is making it so the thread can be restarted in a thread-safe manner and ensure only 1 instance after a reset occurs and more listeners are registered.

[benjchristensen]

Merged commit. Working on one other thing and will then release the next version to Maven Central.

[benjchristensen]

This has been released in 1.2.9.