Correct expectations for cluster connection/command failover. #1757

jdeppe-pivotal · 2021-05-20T18:11:13Z

jdeppe-pivotal
May 20, 2021

Hi.

We're developing a Redis-compatible cluster backend and are using Lettuce and Spring for some integration testing. In one particular case I'm trying to simulate a server failure to understand the ability/limitations of Lettuce to fail over in this scenario. The idea is that the setup should closely mimic a web application. Instead of testing a full web application, I'm creating a SpringApplication context and then using the created SpringRepository to perform operations such as setAttribute, save and findById.

When a backend server fails, other servers in the cluster will automatically start hosting the slots of the departed server.

I have a LettuceConnectionFactory configured as follows:

  @Bean
  public LettuceConnectionFactory connectionFactory(ApplicationArguments applicationArguments) {
    RedisClusterConfiguration redisConfiguration =
        new RedisClusterConfiguration(Arrays.asList(applicationArguments.getSourceArgs()));
    return new LettuceConnectionFactory(redisConfiguration, lettuceClientConfiguration());
  }

  private LettuceClientConfiguration lettuceClientConfiguration() {
    ClusterTopologyRefreshOptions refreshOptions =
        ClusterTopologyRefreshOptions.builder()
            .enableAllAdaptiveRefreshTriggers()
            .enablePeriodicRefresh(Duration.ofSeconds(5))
            .build();

    return LettuceClientConfiguration.builder()
        .clientOptions(ClusterClientOptions.builder()
            .topologyRefreshOptions(refreshOptions)
            .build())
        .build();
  }

During the test, when a server fails while performing saves, I'll often see an exceptions such as:

java.util.concurrent.ExecutionException: org.springframework.data.redis.RedisSystemException: Unknown redis exception; nested exception is java.util.concurrent.CancellationException

	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.geode.redis.session.SessionsAndCrashesDUnitTest.sessionOperationsDoNotFail_whileServersAreRestarted(SessionsAndCrashesDUnitTest.java:178)
	... <snip junit callstack> ...
Caused by: org.springframework.data.redis.RedisSystemException: Unknown redis exception; nested exception is java.util.concurrent.CancellationException
	at org.springframework.data.redis.FallbackExceptionTranslationStrategy.getFallback(FallbackExceptionTranslationStrategy.java:53)
	at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:43)
	at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:274)
	at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.convertLettuceAccessException(LettuceStringCommands.java:800)
	at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.append(LettuceStringCommands.java:455)
	at org.springframework.data.redis.connection.DefaultedRedisConnection.append(DefaultedRedisConnection.java:372)
	at org.springframework.data.redis.core.DefaultValueOperations.lambda$append$5(DefaultValueOperations.java:145)
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:222)
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:189)
	at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:96)
	at org.springframework.data.redis.core.DefaultValueOperations.append(DefaultValueOperations.java:144)
	at org.springframework.data.redis.core.DefaultBoundValueOperations.append(DefaultBoundValueOperations.java:113)
	at org.springframework.session.data.redis.RedisSessionExpirationPolicy.onExpirationUpdated(RedisSessionExpirationPolicy.java:106)
	at org.springframework.session.data.redis.RedisIndexedSessionRepository$RedisSession.saveDelta(RedisIndexedSessionRepository.java:819)
	at org.springframework.session.data.redis.RedisIndexedSessionRepository$RedisSession.save(RedisIndexedSessionRepository.java:783)
	at org.springframework.session.data.redis.RedisIndexedSessionRepository$RedisSession.access$000(RedisIndexedSessionRepository.java:670)
	at org.springframework.session.data.redis.RedisIndexedSessionRepository.save(RedisIndexedSessionRepository.java:398)
	at org.springframework.session.data.redis.RedisIndexedSessionRepository.save(RedisIndexedSessionRepository.java:249)
	at org.apache.geode.redis.session.SessionsAndCrashesDUnitTest.saveSession(SessionsAndCrashesDUnitTest.java:229)
	at org.apache.geode.redis.session.SessionsAndCrashesDUnitTest.sessionUpdater(SessionsAndCrashesDUnitTest.java:210)
	at org.apache.geode.redis.session.SessionsAndCrashesDUnitTest.lambda$sessionOperationsDoNotFail_whileServersAreRestarted$0(SessionsAndCrashesDUnitTest.java:145)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CancellationException
	at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
	at io.lettuce.core.protocol.AsyncCommand.cancel(AsyncCommand.java:146)
	at io.lettuce.core.protocol.AsyncCommand.cancel(AsyncCommand.java:180)
	at io.lettuce.core.protocol.CommandWrapper.cancel(CommandWrapper.java:111)
	at io.lettuce.core.protocol.CommandWrapper.cancel(CommandWrapper.java:111)
	at io.lettuce.core.protocol.DefaultEndpoint.cancelCommands(DefaultEndpoint.java:787)
	at io.lettuce.core.protocol.DefaultEndpoint.cancelBufferedCommands(DefaultEndpoint.java:777)
	at io.lettuce.core.protocol.DefaultEndpoint.closeAsync(DefaultEndpoint.java:603)
	at io.lettuce.core.RedisChannelHandler.closeAsync(RedisChannelHandler.java:152)
	at io.lettuce.core.internal.AsyncConnectionProvider$Sync.doWithConnection(AsyncConnectionProvider.java:278)
	at io.lettuce.core.internal.AsyncConnectionProvider.close(AsyncConnectionProvider.java:181)
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.lambda$closeStaleConnections$7(PooledClusterConnectionProvider.java:542)
	at io.lettuce.core.internal.AsyncConnectionProvider$Sync.doWithConnection(AsyncConnectionProvider.java:287)
	at io.lettuce.core.internal.AsyncConnectionProvider.lambda$forEach$4(AsyncConnectionProvider.java:207)
	at java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1597)
	at io.lettuce.core.internal.AsyncConnectionProvider.forEach(AsyncConnectionProvider.java:207)
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.closeStaleConnections(PooledClusterConnectionProvider.java:540)
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.reconfigurePartitions(PooledClusterConnectionProvider.java:524)
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.setPartitions(PooledClusterConnectionProvider.java:511)
	at io.lettuce.core.cluster.ClusterDistributionChannelWriter.setPartitions(ClusterDistributionChannelWriter.java:471)
	at io.lettuce.core.cluster.StatefulRedisClusterConnectionImpl.setPartitions(StatefulRedisClusterConnectionImpl.java:276)
	at io.lettuce.core.cluster.RedisClusterClient.lambda$updatePartitionsInConnections$23(RedisClusterClient.java:870)
	at io.lettuce.core.cluster.RedisClusterClient.forEachCloseable(RedisClusterClient.java:1105)
	at io.lettuce.core.cluster.RedisClusterClient.forEachClusterConnection(RedisClusterClient.java:1083)
	at io.lettuce.core.cluster.RedisClusterClient.updatePartitionsInConnections(RedisClusterClient.java:869)
	at io.lettuce.core.cluster.RedisClusterClient.lambda$refreshPartitionsAsync$21(RedisClusterClient.java:863)
	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
	at io.lettuce.core.cluster.RedisClusterClient.lambda$loadPartitionsAsync$28(RedisClusterClient.java:927)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
	at io.lettuce.core.protocol.AsyncCommand.completeResult(AsyncCommand.java:122)
	at io.lettuce.core.protocol.AsyncCommand.complete(AsyncCommand.java:111)
	at io.lettuce.core.cluster.topology.TimedAsyncCommand.complete(TimedAsyncCommand.java:52)
	at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:63)
	at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:746)
	at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:681)
	at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:598)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.kqueue.AbstractKQueueStreamChannel$KQueueStreamUnsafe.readReady(AbstractKQueueStreamChannel.java:544)
	at io.netty.channel.kqueue.AbstractKQueueChannel$AbstractKQueueUnsafe.readReady(AbstractKQueueChannel.java:382)
	at io.netty.channel.kqueue.KQueueEventLoop.processReady(KQueueEventLoop.java:211)
	at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:289)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	... 1 more

I appreciate that a save consists of multiple redis commands that are not necessarily transactional so retrying would not be trivial.

I would like to understand the guarantees that Lettuce makes around commands/servers failing and the ability to recover and retry (or not) commands that are currently failing and even those that are currently pending. Is there a succinct way to reason about this? For example, can one assume that failed idempotent operations will always be retried but failed non-idempotent operations (for example append or incrby) would not be retried and would result in an error?

Taking this up a level; if I'm developing a web application (say with Spring Boot, Spring Sessions and Lettuce), backed by a redis cluster, do I have to provide retry logic at the application level for failed operations or will one of those components automatically handle it? (Here I'm constraining 'failed operations' to mean session-specific operations).

Apologies if this is a bit vague, but any insights and pointers would be greatly appreciated. Thanks!

Answered by mp911de

May 21, 2021

Failover is a quite broad topic, so let me add a bit of my perspective here.

Redis Cluster doesn't provide an active notification mechanism for cluster reconfiguration, instead, a client can either poll the cluster topology or react to certain events. Topology polling is available in Lettuce through periodic topology refresh. Other events are modeled as adaptive refresh triggers.

We generally assume if a node is down, that it will eventually come back again (because it has crashed, network partition). We do not assume that it was removed from the cluster in the first place. Therefore, commands sent to a node (either by slot routing or because it was manually routed there) stick with the t…

View full answer

mp911de · 2021-05-21T06:30:27Z

mp911de
May 21, 2021
Maintainer

Failover is a quite broad topic, so let me add a bit of my perspective here.

Redis Cluster doesn't provide an active notification mechanism for cluster reconfiguration, instead, a client can either poll the cluster topology or react to certain events. Topology polling is available in Lettuce through periodic topology refresh. Other events are modeled as adaptive refresh triggers.

We generally assume if a node is down, that it will eventually come back again (because it has crashed, network partition). We do not assume that it was removed from the cluster in the first place. Therefore, commands sent to a node (either by slot routing or because it was manually routed there) stick with the target node until it either comes back online or gets removed from the topology.

If a node comes back up, buffered commands (that didn't time out yet) are retried on the same node.

If a node gets removed from the cluster, then there's a subtle difference in command handling. If a command was sent to a node using its nodeId, then the command gets removed because we assume that using the nodeId was intentional to send a command to a very specific node and we do not want to resend the command to a different node.

If a command was sent to the node using host/port (default mechanism for command routing) and the node gets removed, then we retry/resend the command through the command routing to potentially hit a different server because we assume that ReadFrom server selection should be used to select a target node.

Note that Lettuce command retries are driven by I/O problems only. If a command fails because of a Redis error response, then the command lifecycle was still completed successfully as it has received a response.

Let me know whether that helps and whether you want to discuss further aspects.

1 reply

jdeppe-pivotal May 26, 2021
Author

In the case of I/O errors and commands being retried, is there any distinction in what type of command is executed? Is it not possible for a command to have 'taken effect' but the response doesn't make it back to the client because the server crashed prematurely.

jdeppe-pivotal · 2021-06-08T14:30:58Z

jdeppe-pivotal
Jun 8, 2021
Author

I'm performing a simpler test using only Lettuce. The test crashes a node and then attempts to perform GETs. Since you had said:

If a command was sent to the node using host/port (default mechanism for command routing) and the node gets removed,
then we retry/resend the command through the command routing to potentially hit a different server because we assume that ReadFrom server selection should be used to select a target node.

I was expecting the command to be retried. However, I see that Lettuce continues to attempt to contact the failed node:

[info 2021/06/08 07:08:44.710 PDT  <lettuce-eventExecutorLoop-1-5> tid=0x48] Reconnecting, last destination was 127.0.0.1:60014

[warn 2021/06/08 07:08:44.712 PDT  <lettuce-kqueueEventLoop-4-1> tid=0x32] Cannot reconnect to [127.0.0.1:60014]: finishConnect(..) failed: Connection refused: /127.0.0.1:60014

At some point, it gives up and does a topology refresh and then the following exception is thrown:

java.util.concurrent.CancellationException
	at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
	at io.lettuce.core.protocol.AsyncCommand.cancel(AsyncCommand.java:146)
	at io.lettuce.core.protocol.AsyncCommand.cancel(AsyncCommand.java:180)
	at io.lettuce.core.protocol.CommandWrapper.cancel(CommandWrapper.java:111)
	at io.lettuce.core.protocol.DefaultEndpoint.cancelCommands(DefaultEndpoint.java:787)
	at io.lettuce.core.protocol.DefaultEndpoint.cancelBufferedCommands(DefaultEndpoint.java:777)
	at io.lettuce.core.protocol.DefaultEndpoint.closeAsync(DefaultEndpoint.java:603)
	at io.lettuce.core.RedisChannelHandler.closeAsync(RedisChannelHandler.java:152)
	at io.lettuce.core.internal.AsyncConnectionProvider$Sync.doWithConnection(AsyncConnectionProvider.java:278)
	at io.lettuce.core.internal.AsyncConnectionProvider.close(AsyncConnectionProvider.java:181)
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.lambda$closeStaleConnections$7(PooledClusterConnectionProvider.java:542)
	at io.lettuce.core.internal.AsyncConnectionProvider$Sync.doWithConnection(AsyncConnectionProvider.java:287)
	at io.lettuce.core.internal.AsyncConnectionProvider.lambda$forEach$4(AsyncConnectionProvider.java:207)
	at java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1597)
	at io.lettuce.core.internal.AsyncConnectionProvider.forEach(AsyncConnectionProvider.java:207)
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.closeStaleConnections(PooledClusterConnectionProvider.java:540)
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.reconfigurePartitions(PooledClusterConnectionProvider.java:524)
	at io.lettuce.core.cluster.PooledClusterConnectionProvider.setPartitions(PooledClusterConnectionProvider.java:511)
	at io.lettuce.core.cluster.ClusterDistributionChannelWriter.setPartitions(ClusterDistributionChannelWriter.java:471)
	at io.lettuce.core.cluster.StatefulRedisClusterConnectionImpl.setPartitions(StatefulRedisClusterConnectionImpl.java:276)
	at io.lettuce.core.cluster.RedisClusterClient.lambda$updatePartitionsInConnections$23(RedisClusterClient.java:870)
	at io.lettuce.core.cluster.RedisClusterClient.forEachCloseable(RedisClusterClient.java:1105)
	at io.lettuce.core.cluster.RedisClusterClient.forEachClusterConnection(RedisClusterClient.java:1083)
	at io.lettuce.core.cluster.RedisClusterClient.updatePartitionsInConnections(RedisClusterClient.java:869)
	at io.lettuce.core.cluster.RedisClusterClient.lambda$refreshPartitionsAsync$21(RedisClusterClient.java:863)
	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire$$$capture(CompletableFuture.java:646)
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
	at io.lettuce.core.cluster.RedisClusterClient.lambda$loadPartitionsAsync$28(RedisClusterClient.java:927)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
	at io.lettuce.core.protocol.AsyncCommand.completeResult(AsyncCommand.java:122)
	at io.lettuce.core.protocol.AsyncCommand.complete(AsyncCommand.java:111)
	at io.lettuce.core.cluster.topology.TimedAsyncCommand.complete(TimedAsyncCommand.java:52)
	at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:63)
	at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:746)
	at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:681)
	at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:598)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.kqueue.AbstractKQueueStreamChannel$KQueueStreamUnsafe.readReady(AbstractKQueueStreamChannel.java:544)
	at io.netty.channel.kqueue.AbstractKQueueChannel$AbstractKQueueUnsafe.readReady(AbstractKQueueChannel.java:382)
	at io.netty.channel.kqueue.KQueueEventLoop.processReady(KQueueEventLoop.java:211)
	at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:289)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

and the command is not retried.

5 replies

mp911de Jun 8, 2021
Maintainer

Thanks for having a deeper look. Let me check what's going on since my expectation was also a different one.

mp911de Jun 9, 2021
Maintainer

This is actually a bug that is in place since 2017. With the introduction and refactoring to asynchronous connect methods, we missed migrating a method override. Converting this discussion into a bug report.

mp911de Jun 9, 2021
Maintainer

That's fixed now with #1769

jdeppe-pivotal Jun 9, 2021
Author

Nice. Thanks for fixing this so quickly!!

jdeppe-pivotal Jun 9, 2021
Author

I can also confirm that this fixes the scenario I was testing.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Correct expectations for cluster connection/command failover. #1757

{{title}}

Replies: 2 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Correct expectations for cluster connection/command failover. #1757

jdeppe-pivotal May 20, 2021

Replies: 2 comments · 6 replies

mp911de May 21, 2021 Maintainer

jdeppe-pivotal May 26, 2021 Author

jdeppe-pivotal Jun 8, 2021 Author

mp911de Jun 8, 2021 Maintainer

mp911de Jun 9, 2021 Maintainer

mp911de Jun 9, 2021 Maintainer

jdeppe-pivotal Jun 9, 2021 Author

jdeppe-pivotal Jun 9, 2021 Author

jdeppe-pivotal
May 20, 2021

Replies: 2 comments 6 replies

mp911de
May 21, 2021
Maintainer

jdeppe-pivotal May 26, 2021
Author

jdeppe-pivotal
Jun 8, 2021
Author

mp911de Jun 8, 2021
Maintainer

mp911de Jun 9, 2021
Maintainer

mp911de Jun 9, 2021
Maintainer

jdeppe-pivotal Jun 9, 2021
Author

jdeppe-pivotal Jun 9, 2021
Author