Lots of boundedElastic-evictor TIMED_WAITING threads related to form data requests

[Justubborn]

I send request to webflux demo server. use springboot version 2.4.1 , create no quantity limit nuboundedElastic-evictor thread and always keep timed waiting status.

test script

ab -n 10000 -c 1000 -T "application/x-www-form-urlencoded" -p test.txt http://127.0.0.1:8080/router/rest/1

controller code

@RestController
@RequestMapping("/router/rest")
public class BaseController {

    @RequestMapping("1")
    public Mono<String> mono(){
        return Mono.just("111");
    }
}

lots of boundedElastic-evictor TIMED_WAITING make memory up obvious.

"boundedElastic-evictor-9998" #10042 daemon prio=5 os_prio=0 cpu=0.00ms elapsed=130.64s tid=0x0000023b82f71dc0 nid=0xe694 waiting on condition  [0x0000008cf91fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
        - parking to wait for  <0x000000070aba6988> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:252)

finally

Switching to version 2.3.7 resolved

[rstoyanchev]

Can you provide more details around "no quantity limit nuboundedElastic-evictor thread"? I'm also not sure what you mean by "make memory up obvious".

[Justubborn]

Can you provide more details around "no quantity limit nuboundedElastic-evictor thread"? I'm also not sure what you mean by "make memory up obvious".

after send 1000 request ,the program create 1000 thread name "boundedElastic-evictor" and java memory from 50M increase to 500M

[jun-lee]

+1
Spring boot 2.4.0 + Webflux has just the same issue.

[rstoyanchev]

Can you provide more details around "no quantity limit nuboundedElastic-evictor thread"?

I still need this. For example a snippet of how you configure this.

[rstoyanchev]

Actually is it possible to provide a sample app? It's unclear where the boundedElastic use comes from.

[Justubborn]

Actually is it possible to provide a sample app? It's unclear where the boundedElastic use comes from.

Below is all the code, no additional configuration：

@RestController
@RequestMapping("/router/rest")
@SpringBootApplication
public class DemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }

    @RequestMapping("1")
    public Mono<String> mono(){
        return Mono.just("111");
    }
}

Below is the thread stack. You can tell from the stack log where boundedElastic came from.

"reactor-http-nio-2@5381" daemon prio=5 tid=0x2a nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at reactor.core.scheduler.BoundedElasticScheduler.lambda$static$0(BoundedElasticScheduler.java:75)
	  at reactor.core.scheduler.BoundedElasticScheduler$$Lambda$382.45019084.newThread(Unknown Source:-1)
	  at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619)
	  at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
	  at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603)
	  at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
	  at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:573)
	  at reactor.core.scheduler.BoundedElasticScheduler.start(BoundedElasticScheduler.java:175)
	  at reactor.core.scheduler.Schedulers.newBoundedElastic(Schedulers.java:498)
	  at reactor.core.scheduler.Schedulers.newBoundedElastic(Schedulers.java:454)
	  at org.springframework.http.codec.multipart.DefaultPartHttpMessageReader.<init>(DefaultPartHttpMessageReader.java:77)
	  at org.springframework.http.codec.support.ServerDefaultCodecsImpl.extendTypedReaders(ServerDefaultCodecsImpl.java:72)
	  at org.springframework.http.codec.support.BaseDefaultCodecs.getTypedReaders(BaseDefaultCodecs.java:275)
	  at org.springframework.http.codec.support.BaseCodecConfigurer.getReaders(BaseCodecConfigurer.java:98)
	  at org.springframework.http.codec.support.DefaultServerCodecConfigurer.getReaders(DefaultServerCodecConfigurer.java:27)
	  at org.springframework.web.server.adapter.DefaultServerWebExchange.initFormData(DefaultServerWebExchange.java:143)
	  at org.springframework.web.server.adapter.DefaultServerWebExchange.<init>(DefaultServerWebExchange.java:131)
	  at org.springframework.web.server.adapter.HttpWebHandlerAdapter.createExchange(HttpWebHandlerAdapter.java:243)
	  at org.springframework.web.server.adapter.HttpWebHandlerAdapter.handle(HttpWebHandlerAdapter.java:229)
	  at org.springframework.boot.web.reactive.context.WebServerManager$DelayedInitializationHttpHandler.handle(WebServerManager.java:97)
	  at org.springframework.http.server.reactive.ReactorHttpHandlerAdapter.apply(ReactorHttpHandlerAdapter.java:65)
	  at org.springframework.http.server.reactive.ReactorHttpHandlerAdapter.apply(ReactorHttpHandlerAdapter.java:40)
	  at reactor.netty.http.server.HttpServer$HttpServerHandle.onStateChange(HttpServer.java:628)
	  at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:612)
	  at reactor.netty.transport.ServerTransport$ChildObserver.onStateChange(ServerTransport.java:453)
	  at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:510)
	  at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94)
	  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	  at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:208)
	  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	  at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	  at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
	  at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311)
	  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432)
	  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	  at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
	  at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
	  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
	  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	  at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	  at java.lang.Thread.run(Thread.java:748)

[jun-lee]

Ok, I found the root cause.

spring-framework/spring-web/src/main/java/org/springframework/http/codec/support/ServerDefaultCodecsImpl.java

Line 72 in 499be70

DefaultPartHttpMessageReader partReader = new DefaultPartHttpMessageReader();

spring-framework/spring-web/src/main/java/org/springframework/http/codec/multipart/DefaultPartHttpMessageReader.java

Line 77 in 499be70

private Scheduler blockingOperationScheduler = Schedulers.newBoundedElastic(Schedulers.DEFAULT_BOUNDED_ELASTIC_SIZE,

From the above points, a Http request has a Content-Type: "application/x-www-form-urlencoded" causes boundedElastic-evictor-# garbage everytime.

[jun-lee]

A quick workaround would be adding a config.

@Configuration
@EnableWebFlux
public class WebConfig implements WebFluxConfigurer {

    @Override
    public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
        DefaultPartHttpMessageReader multipartReader = new DefaultPartHttpMessageReader();
        configurer.defaultCodecs().multipartReader(multipartReader);
    }

}

[rstoyanchev]

Thanks for the extra details. This is a combination of DefaultPartHttpMessageReader creating resources internally and DefaultServerWebExchange re-creating the codecs per request. We need to make this more efficient and also ensure those resources support a full lifecycle with start and stop.