Unable to connect to AWS ElastiCache Redis after few days of successful working

[srikarrampa]

Bug Report

Observing that lettuce client fails to process commands successfully after 3-4 days of successful deployments and working.
The current client options are

TimeoutOptions timeoutOptions =
        TimeoutOptions.builder().fixedTimeout(Duration.ofMillis(1000)).build();

    ClientOptions clientOptions =
        ClientOptions.builder()
            .socketOptions(
                SocketOptions.builder()
                    .connectTimeout(Duration.ofMillis(1000))
                    .build())
            .timeoutOptions(timeoutOptions)
            .autoReconnect(true))
            .suspendReconnectOnProtocolFailure(true)
            .disconnectedBehavior(ClientOptions.DisconnectedBehavior.REJECT_COMMANDS)
            .build();

    ClientResources clientResources =
        DefaultClientResources.builder()
            .dnsResolver(new DirContextDnsResolver())
            .reconnectDelay(
                Delay.fullJitter(
                    Duration.ofMillis(300),
                    Duration.ofMillis(700),
                    redisConfig.getBase(),
                    TimeUnit.MILLISECONDS))
            .build();
    LettuceClientConfiguration.LettuceClientConfigurationBuilder builder =
        LettuceClientConfiguration.builder()
            .readFrom(ReadFrom.REPLICA_PREFERRED)
            .clientOptions(clientOptions)
            .clientResources(clientResources)
            .redisCredentialsProviderFactory(redisCredentialsProviderFactory());

Current Behavior

Stack trace

java.util.concurrent.CompletionException: io.lettuce.core.RedisCommandExecutionException: WRONGPASS invalid username-password pair or user is disabled.
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1141)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
	at io.lettuce.core.RedisHandshake.lambda$tryHandshakeResp3$2(RedisHandshake.java:134)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
	at io.lettuce.core.protocol.AsyncCommand.doCompleteExceptionally(AsyncCommand.java:143)
	at io.lettuce.core.protocol.AsyncCommand.completeResult(AsyncCommand.java:124)
	at io.lettuce.core.protocol.AsyncCommand.complete(AsyncCommand.java:115)
	at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:67)
	at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:762)
	at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:697)
	at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:614)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1503)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1366)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1415)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: io.lettuce.core.RedisCommandExecutionException: WRONGPASS invalid username-password pair or user is disabled.
	at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:151)
	at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:120)
	... 29 more

Error message: Disabling autoReconnect due to initialization failure

We see the above error and after that all the requests are rejected. Is there some issue in the configuration settings? The process we follow for creating a connection with AWS ElastiCache is as follows,

• Assume a role that provides with SessionToken
• Pass that to a Class that extends RedisCredentialsProvider which resolvesCredentials (public Mono resolveCredentials())

Input Code

Input Code

// your code here;

Expected behavior/code

The reconnection should happen

Environment

• Lettuce version(s): 6.5.1.RELEASE
• Redis version: 7.1.0

Questions

• What is the correct ClientOptions to use in production to allow re-connect. Is it advised to have all three together?
  .autoReconnect(true))
  .suspendReconnectOnProtocolFailure(true)
  .disconnectedBehavior(ClientOptions.DisconnectedBehavior.REJECT_COMMANDS)

• Based on the above settings, reconnect is suspended. how should we recover from this?

• Restarting the deployment fixed the issue. However we cannot restart the deployment everytime. Please do provide if any documentation that explains the best practices to connect when using AWS ElastiCache