Remove virtual thread support for Undertow as it leaks memory #39812
Comments
spring-projects-issues
added
the
status: waiting-for-triage
|
That's one of the drawbacks when moving from a thread pool to virtual threads per task without a pool. When using thread pools, they usually have an upper bound for the size, limiting the maximum resource consumption. Unpooled virtual threads don't have that. I assume the application you're benchmarking is a simple demo application which just returns some hardcoded data? Because usually, if there's no pool in the webserver, some other pool in the application is limiting the throughput (e.g. the connection pool to the database). Btw, you don't see the memory usage in VisualVM, as this is direct allocated memory, which is off-heap. |
mhalbritter
added
the
status: waiting-for-feedback
|
Yes, it's the Spring Boot demo project with a basic GET endpoint with no return value. Sorry, could you explain the cause for this again? Aren't virtual threads still limited by the size of the platform threads in the carrier pool? The Tomcat and Jetty servlets don't seem to have the same issue when using virtual threads. Also, thank you for the explanation about the profiler. |
spring-projects-issues
added
status: feedback-provided
|
One possibility could be this: With enough requests lined up, the block at 3. could be the reason why the memory is exhausted. At this block, the scheduler frees the underlying platform thread, and switches to a new virtual thread, which allocates memory, blocks, a new virtual thread is scheduled, allocates memory, etc etc. |
|
You could use a |
|
as long as I remember, there is an issue that talks about virtual threads in Spring Boot here: there is one part I saw that says in version 3.3.x the usage of the virtual thread must automatically happen for undertow. so can it be related to this issue? can u check if this also happens with version 3.2.x ? @Tythor |
|
@MetaiR @mhalbritter @Bean
public UndertowDeploymentInfoCustomizer undertowDeploymentInfoCustomizer() {
return deploymentInfo -> {
SimpleAsyncTaskExecutor simpleAsyncTaskExecutor = new SimpleAsyncTaskExecutor();
simpleAsyncTaskExecutor.setVirtualThreads(true);
simpleAsyncTaskExecutor.setConcurrencyLimit(10);
deploymentInfo.setExecutor(simpleAsyncTaskExecutor);
};
}However, this led me to try using a non virtual threaded executor instead. Surprisingly, I observed the same behavior, albeit at a much slower growth rate of about 2m for 20GB instead of 30s. These three executors produced similar results: deploymentInfo.setExecutor(new SimpleAsyncTaskExecutor());
deploymentInfo.setExecutor(Executors.newThreadPerTaskExecutor(Thread.ofPlatform().factory()));
deploymentInfo.setExecutor(Executors.newThreadPerTaskExecutor(Executors.defaultThreadFactory()));However, when using deploymentInfo.setExecutor(Executors.newCachedThreadPool());the memory growth disappeared. Looking into the implementation, the main difference is that the cachedThreadPool executor has a Reducing the deploymentInfo.setExecutor(new ThreadPoolExecutor(0, Integer.MAX_VALUE, 1L, TimeUnit.SECONDS, new SynchronousQueue<>()));So it looks like this issue is not unique to virtual threads, but perhaps with the way Undertow is handling executors? |
|
Yeah, looks like something is wrong here, but it doesn't seem to be in Spring Boot. Please open an issue on the Undertow tracker, and feel free to drop the link in this issue. Thanks! |
mhalbritter
added
status: invalid
|
We decided to reopen the issue. The out of box experience with virtual threads in undertow is bad, and if we (or the undertow team) can't fix the problems with it, we might remove virtual threads support for undertow again. |
mhalbritter
changed the title
|
Okay, I was about to suggest disabling the auto configuration for |
|
Yeah, we shouldn't fiddle with the Undertow executors. The default one produces this plot:
As soon as we meddle with it, it starts breaking. With virtual threads:
With virtual threads and concurrency throttle to 200:
With platform threads and concurrency throttle to 200:
It breaks in all configurations when setting the executor. I'm going to revert the virtual thread support for Undertow. |
mhalbritter
added
type: bug
mhalbritter
changed the title
wilkinsona
added
the
status: noteworthy
|
It looks like Undertow may have a memory leak. @Tythor, can you please retry your scenario with Undertow downgraded to 2.3.10.Final and see if it helps? |
|
Hi @wilkinsona, downgrading Undertow to 2.3.10.Final on Spring Boot v3.3.0-M2 did significantly slow the memory growth. However, I am still seeing suspicious behavior when providing an executor in the |
|
Undertow 2.3.10.Final,
It's better, but there's still a problem somewhere. |
|
This is what 2.3.10 with the default executor looks like:
|
|
Thank you @mhalbritter, these plots are very similar to the results I observed as well. |
|
Btw, the tool I used is https://github.com/astrofrog/psrecord. |
|
Hello, I am using Spring Boot 3.2.4 and Undertow with OpenJDK by Corretto, I don't have this issue. My application is an API REST, it is running by several days. |
|
@cassiusvm How have you configured Undertow to use virtual threads? |
|
@wilkinsona Right, using the @configuration tip in a Docker image jelastic maven 3.9.5 OpenJDK 21. My bad, I said Correto. |
|
I realise its not a spring issue per-se, but hoping that this is useful. When configuring with The service will quickly blow up.. I think due to use of ThreadLocal buffer strategy.
In It is possible to make it used unpooled buffers, with but the performance of this is worse (by 50%) than the default, so no point. Anyhow, hope thats of some use if somebody is looking into this. |
Allows enabling virtual thread support for Jetty and Tomcat. Undertow has issues with Virtual threads so not implemented. See spring-projects/spring-boot#39812 Co-authored-by: Sergio del Amo <[email protected]> --------- Co-authored-by: Sergio del Amo <[email protected]>
|
|
|
Thanks @JavaLionLi. I'll update #38819 with that info. |
Hello,
I am observing an issue with virtual threads in Undertow in Spring Boot v3.3.0-M2. When using
spring.threads.virtual.enabled=truein the application.properties oras a configuration, the memory usage of the application increases dramatically under load. I performed a wrk test with the command
wrk -d30 -t10 -c150 http://localhost:8080and my application's memory usage increased to 20GB in just 30 seconds.Interestingly, when using a profiler, I observed that the JVM reported significantly lower memory usage than what the activity monitor was showing.
The application eventually crashes with an OOM error:
Output of
java -versionI was able to reproduce this behavior when running the application in a Docker container, when compiling to a native-image, on different JDKs (OpenJDK, GraalVM, and Corretto), and on different macOS architectures (Intel and Apple silicon).
Tagging related issue #38819.
Thank you.
The text was updated successfully, but these errors were encountered: