OpsForGeo producing "READONLY You can't write against a read only replica" on READS... only if master & replica configured

[boxingnight]

Bug Report

This is a strange one and something that is causing us to depend on a single master cache instance for both our reads and our writes.

Current Behavior

When we configure Lettuce with the configuration below, we see it produce the following stacktrace when we perform a read operation using:

redisTemplate.opsForGeo().radius(...)

If we only configure a single master instance using RedisStandaloneConfiguration we do not see this issue at all.

Stack trace

2021-07-20 13:58:27.904 ERROR 37538 --- [nio-8080-exec-4] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.data.redis.RedisSystemException: Error in execution; nested exception is io.lettuce.core.RedisCommandExecutionException: READONLY You can't write against a read only replica.] with root cause

io.lettuce.core.RedisCommandExecutionException: READONLY You can't write against a read only replica.
	at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:137) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
	at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:110) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
	at io.lettuce.core.protocol.AsyncCommand.completeResult(AsyncCommand.java:120) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
	at io.lettuce.core.protocol.AsyncCommand.complete(AsyncCommand.java:111) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
	at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:720) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
	at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:655) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
	at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:572) ~[lettuce-core-6.0.2.RELEASE.jar:6.0.2.RELEASE]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]

Input Code

Configuration

@Bean  
public RedisStaticMasterReplicaConfiguration redisConfiguration() {  
  
    RedisStaticMasterReplicaConfiguration redisStaticMasterReplicaConfiguration  
  = new RedisStaticMasterReplicaConfiguration(properties.getPrimaryEndpoint());  
  
    redisStaticMasterReplicaConfiguration.node(properties.getReaderEndpoint());  
  
    return redisStaticMasterReplicaConfiguration;  
}  
  
@Bean  
public LettuceClientConfiguration lettuceClientConfiguration() {  
  
    ClientOptions clientOptions = ClientOptions.builder()  
            .autoReconnect(true)  
            .build();  
  
    return LettuceClientConfiguration.builder()  
            .readFrom(ReadFrom.REPLICA_PREFERRED)  
            .clientOptions(clientOptions)  
            .build();  
}  
  
@Bean  
public RedisTemplate<String, Object> redisTemplate(RedisStaticMasterReplicaConfiguration redisConfiguration,  
                                   LettuceClientConfiguration lettuceClientConfiguration) {  
  
    LettuceConnectionFactory lettuceConnectionFactory =  
            new LettuceConnectionFactory(redisConfiguration, lettuceClientConfiguration);  
    lettuceConnectionFactory.afterPropertiesSet();  
  
    RedisTemplate<String, Object> redisTemplate = new RedisTemplate<>();  
    redisTemplate.setConnectionFactory(lettuceConnectionFactory);  
    redisTemplate.setEnableTransactionSupport(true);  
    redisTemplate.setKeySerializer(new StringRedisSerializer());  
    redisTemplate.setValueSerializer(new JdkSerializationRedisSerializer());  
    redisTemplate.afterPropertiesSet();  
  
    return redisTemplate;  
}

Expected behavior/code

GeoResults<RedisGeoCommands.GeoLocation<Object>> result =  
        redisTemplate.opsForGeo().radius(PREFIX + myKey,  
                circle);

Should produce an empty/non-empty results object.

Environment

• Lettuce as part of Spring Boot & Data 2.4.2
• AWS - One primary, One read replica

Additional context

If you break point in the code and execute the radius() command numerous times within the same process, it fails every few times. If, however, you execute it via a REST endpoint, it will fail every time.

[mp911de]

This comes from the nature that GEORADIUS is a command flagged as write-command. For Redis Cluster, Lettuce diverts pure read intentions (variants that do not use STORE/STOREDIST) to GEORADIUS_RO. It would make sense to use a similar approach for Redis Standalone/Replica arrangements.

Output of the COMMAND command:

1) 1) "georadius_ro"
     2) (integer) -6
     3) 1) "readonly"
     4) (integer) 1
     5) (integer) 1
     6) (integer) 1
     7) 1) "@read"
        2) "@geo"
        3) "@slow"


173) 1) "georadius"
     2) (integer) -6
     3) 1) "write"
        2) "denyoom"
        3) "movablekeys"
     4) (integer) 1
     5) (integer) 1
     6) (integer) 1
     7) 1) "@write"
        2) "@geo"
        3) "@slow"