java.util.concurrent.RejectedExecutionException: event executor terminated

[sadxiaohu]

Bug Report

Hi , this is Hu , I want to report a bug of lettuce . I used lettuce to monitor status of Redis Cluster(like cluster status , masters ,slaves etc) , it usually works well , but it may have data point missing issue suddenly .

Current Behavior

Monitoring Service may not get information of Redis Cluster due to unknown reason.

Stack trace

02 Nov 2020 09:32:53,270 [WARN] e1ee7a7a-f178-4f6a-8825-b4219909cf40 (pool-12-thread-35) io.netty.channel.AbstractChannel: Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xd9ffe45a]
java.util.concurrent.RejectedExecutionException: event executor terminated
    at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926)
    at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353)
    at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346)
    at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828)
    at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:471)
    at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:87)
    at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81)
    at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
    at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:323)
    at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:155)
    at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:139)
    at io.lettuce.core.AbstractRedisClient.initializeChannelAsync0(AbstractRedisClient.java:313)
    at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync$0(AbstractRedisClient.java:294)
    at reactor.core.publisher.LambdaMonoSubscriber.onNext(LambdaMonoSubscriber.java:168)
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203)
    at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173)
    at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onNext(FluxPeekFuseable.java:845)
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1637)
    at reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:61)
    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4105)
    at reactor.core.publisher.Mono.subscribeWith(Mono.java:4211)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4077)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4013)
    at reactor.core.publisher.Mono.subscribe(Mono.java:3985)
    at io.lettuce.core.AbstractRedisClient.initializeChannelAsync(AbstractRedisClient.java:289)
    at io.lettuce.core.RedisClient.connectStatefulAsync(RedisClient.java:294)
    at io.lettuce.core.RedisClient.connectStandaloneAsync(RedisClient.java:274)
    at io.lettuce.core.RedisClient.connect(RedisClient.java:207)
    at io.lettuce.core.RedisClient.connect(RedisClient.java:192)
    at com.amazon.redisnanny.provider.LettuceClient.getMonitorParameters(LettuceClient.java:81)
    at com.amazon.redisnanny.provider.ReplicationInfoCollector.lambda$getCurrentSnapshot$3(ReplicationInfoCollector.java:90)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:749)
02 Nov 2020 09:32:53,271 [ERROR] e1ee7a7a-f178-4f6a-8825-b4219909cf40 (pool-12-thread-35) io.netty.util.concurrent.DefaultPromise.rejectedExecution: Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
    at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926)
    at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353)
    at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346)
    at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828)
    at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
    at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:498)
    at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:183)
    at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95)
    at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30)
    at io.lettuce.core.AbstractRedisClient.initializeChannelAsync0(AbstractRedisClient.java:336)
    at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync$0(AbstractRedisClient.java:294)
    at reactor.core.publisher.LambdaMonoSubscriber.onNext(LambdaMonoSubscriber.java:168)
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203)
    at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173)
    at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onNext(FluxPeekFuseable.java:845)
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1637)
    at reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:61)
    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4105)
    at reactor.core.publisher.Mono.subscribeWith(Mono.java:4211)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4077)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4013)
    at reactor.core.publisher.Mono.subscribe(Mono.java:3985)
    at io.lettuce.core.AbstractRedisClient.initializeChannelAsync(AbstractRedisClient.java:289)
    at io.lettuce.core.RedisClient.connectStatefulAsync(RedisClient.java:294)
    at io.lettuce.core.RedisClient.connectStandaloneAsync(RedisClient.java:274)
    at io.lettuce.core.RedisClient.connect(RedisClient.java:207)
    at io.lettuce.core.RedisClient.connect(RedisClient.java:192)
    at com.amazon.redisnanny.provider.LettuceClient.getMonitorParameters(LettuceClient.java:81)
    at com.amazon.redisnanny.provider.ReplicationInfoCollector.lambda$getCurrentSnapshot$3(ReplicationInfoCollector.java:90)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:749)

Thread dump

"RedisConsistencyMonitor-0" #57 prio=5 os_prio=0 tid=0x00007fbcf9996000 nid=0x2bb3 waiting on condition [0x00007fbc5abf0000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000722e02378> (a java.util.concurrent.FutureTask)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
    at java.util.concurrent.FutureTask.get(FutureTask.java:191)
    at com.amazon.redisnanny.provider.ReplicationInfoCollector.getCurrentSnapshot(ReplicationInfoCollector.java:103)
    at com.amazon.redisnanny.provider.ReplicationInfoCollector.getReplicationInfo(ReplicationInfoCollector.java:48)
    at com.amazon.redisnanny.admin.RedisConsistencyMonitor.metricsCollect(RedisConsistencyMonitor.java:136)
    at com.amazon.redisnanny.admin.RedisConsistencyMonitor$$Lambda$37/756222660.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:749)

#### Input Code

Input Code

for (String host : hostNames) {
    
    ...

    long startCollect = System.currentTimeMillis();
    RedisClient client = null;
    StatefulRedisConnection<String, String> connection = null;
    ...
    try {
        client = RedisProvider.getRedisClient(redisProvider.generateRedisURI(host));
        connection = client.connect();

        ...

    } catch (Exception e) {
        ...
        ...
    } finally {
        if (connection != null) {
            connection.close();
        }
        if (client != null) {
            client.shutdown();
        }
        ...
    }
}
......

lettuce code in RedisClient class

    public StatefulRedisConnection<String, String> connect() {
        return this.connect(this.newStringStringCodec());
    }

Expected behavior/code

should be no issue when connect to Redis Cluster host

Environment

• Lettuce version(s): 5.3.1.RELEASE
• Redis version: 5.0.4

Possible Solution

We do not find the root cause of this issue ,so that we do not know how to reslove it.

Additional context

We can not reproduce it , but it happens for several times.

[mp911de]

RejectedExecutionException is a consequence of shutting down the EventLoop (ClientResources.shutdown(), RedisClient.shutdown()). Please ensure that you close the event loop after completing all outstanding Redis commands.

[sadxiaohu]

RejectedExecutionException is a consequence of shutting down the EventLoop (ClientResources.shutdown(), RedisClient.shutdown()). Please ensure that you close the event loop after completing all outstanding Redis commands.

Hello , this information is very useful for us .
However , our code logic is very straightforward as you can see in input code part . Besides , it's scheduled job , and it will be executed every 15 seconds . It looks we never use Redis Commands after we shutdown RedisClient.

Can you help analyze it? We are stuck for over 1 month.
One information you may need: when we create a Redis Client , we use ClientResource like this:

CLIENT_RESOURCES = DefaultClientResources.builder()
                .commandLatencyCollectorOptions(DefaultCommandLatencyCollectorOptions.disabled())
                .build();
client = RedisClient.create(CLIENT_RESOURCES, redisProvider.generateRedisURI(host));

Also , this thread is blocked ,and no more work will execute.

Hope to get response from you

[445990772]

hello, I have the same problem, and I set up a redis cluster with three master nodes no slave nodes。

redis version :5.0.4
this is thread stack exception info:

2024-02-22 11:44:05.676 DEBUG 6 --- [or-http-epoll-8] o.h.web.crud.web.ResponseMessageWrapper  : [84981626-2761] 0..1 [org.hswebframework.web.crud.web.ResponseMessage<?>]
2024-02-22 11:44:05.741  WARN 6 --- [llEventLoop-6-5] io.lettuce.core.protocol.CommandHandler  : null Unexpected exception during request: java.util.concurrent.RejectedExecutionException: event executor terminated

java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934)
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351)
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344)
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836)
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)
        at io.lettuce.core.RedisPublisher$PublishOnSubscriber.onError(RedisPublisher.java:934)
        at io.lettuce.core.RedisPublisher$State.onError(RedisPublisher.java:712)
        at io.lettuce.core.RedisPublisher$RedisSubscription.onError(RedisPublisher.java:357)
        at io.lettuce.core.RedisPublisher$RedisSubscription.onNext(RedisPublisher.java:293)
        at io.lettuce.core.RedisPublisher$SubscriptionCommand.doOnComplete(RedisPublisher.java:773)
        at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:65)
        at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:63)
        at io.lettuce.core.cluster.ClusterCommand.complete(ClusterCommand.java:65)
        at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:63)
        at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:747)
        at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:682)
        at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:599)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

this is jstack info :
iot-20240222-01.txt

this is my appliction jvm settings: -Xms8G -Xmx8G -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
redis config used default settings.

[mp911de]

RejectedExecutionException: event executor terminated means that the underlying thread pool is stopped while the Redis Client was connected and still active running tasks.