Unable to connect to [172.25.0.4/<unresolved>:6379]

[kioritanaka]

I am spinning up a Redis cluster locally using this docker-compose file and running the following command:

redis-cli --cluster create redis-cache-1:6379 redis-cache2-1:6379 redis-cache3-1:6379 --cluster-replicas 0

and trying to switch over to using the cluster-aware client using the following code since we were seeing some MOVED errors in our test environments after enabling cluster mode:

public static class RedisConnection {
    private StatefulRedisClusterConnection<byte[], byte[]> _redisPrimaryConnection;
    private final RedisClusterClient _redisPrimaryClient;

    private RedisConnection(final ServiceSettings serviceSettings) {
      final String redisUsername;
      final String redisPassword;
      if (serviceSettings.getServiceEnv().isLocal() || serviceSettings.getServiceEnv().isTest()) {
        // No Redis creds configured for local or test environment
        redisUsername = "";
        redisPassword = "";
      } else {
        final String userEnvVarName =
            serviceSettings.getString("redis.userCredentials.usernameEnvVarName");
        redisUsername = getSecretFromHelix(serviceSettings, userEnvVarName);
        final String passwordEnvVarName =
            serviceSettings.getString("redis.userCredentials.passwordEnvVarName");
        redisPassword = getSecretFromHelix(serviceSettings, passwordEnvVarName);
      }

      var clusterTopologyRefreshOptions =
          ClusterTopologyRefreshOptions.builder()
              .enableAdaptiveRefreshTrigger(
                  ClusterTopologyRefreshOptions.RefreshTrigger.MOVED_REDIRECT,
                  ClusterTopologyRefreshOptions.RefreshTrigger.PERSISTENT_RECONNECTS)
              .enablePeriodicRefresh(Duration.ofMinutes(30))
              .build();
      final ClusterClientOptions clusterClientOptions =
          ClusterClientOptions.builder()
              .topologyRefreshOptions(clusterTopologyRefreshOptions)
              .build();

      List<RedisURI> redisURIs =
          Arrays.asList(
              RedisURI.Builder.redis("127.0.0.1", 6379).build(),
              RedisURI.Builder.redis("127.0.0.1", 6380).build(),
              RedisURI.Builder.redis("127.0.0.1", 6381).build());

      _redisPrimaryClient = RedisClusterClient.create(redisURIs);
      _redisPrimaryClient.setOptions(clusterClientOptions);
      _redisPrimaryConnection = _redisPrimaryClient.connect(new ByteArrayCodec());

The last line from the above is producing the following error:

[service.id_] 18:30:32.641 ERROR (DnsServerAddressStreamProviders.java: Unable to load io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'. Use DEBUG level to see the full stack: java.lang.UnsatisfiedLinkError: failed to load the required native library
[service.id_] 18:30:42.987 WARN  (DefaultClusterTopologyRefresh.java:34 Unable to connect to [172.25.0.2/<unresolved>:6379]: connection timed out: /172.25.0.2:6379
[service.id_] 18:30:42.988 WARN  (DefaultClusterTopologyRefresh.java:34 Unable to connect to [172.25.0.3/<unresolved>:6379]: connection timed out: /172.25.0.3:6379
[service.id_] 18:30:42.988 WARN  (DefaultClusterTopologyRefresh.java:34 Unable to connect to [172.25.0.4/<unresolved>:6379]: connection timed out: /172.25.0.4:6379
[service.id_] 18:30:53.080 WARN           (RedisClusterClient.java:717) connection timed out: /172.25.0.3:6379
[service.id_] 18:31:03.091 WARN           (RedisClusterClient.java:717) connection timed out: /172.25.0.2:6379
[service.id_] 18:31:13.098 WARN           (RedisClusterClient.java:717) connection timed out: /172.25.0.4:6379
[service.id_] 18:31:13.111 ERROR           (ConnectionProvider.java:66) Failed to create redis connection
io.lettuce.core.RedisConnectionException: Unable to connect
	at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:79)
	at io.lettuce.core.AbstractRedisClient.getConnection(AbstractRedisClient.java:376)
	at io.lettuce.core.cluster.RedisClusterClient.connect(RedisClusterClient.java:407)
	.... (omitted lines)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
	at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113)
	at com.google.inject.Guice.createInjector(Guice.java:87)
	at com.google.inject.Guice.createInjector(Guice.java:69)
	.... (omitted lines)
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /172.25.0.4:6379
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Assembly trace from producer [reactor.core.publisher.MonoCompletionStage] :
	reactor.core.publisher.Mono.fromCompletionStage(Mono.java:543)
	io.lettuce.core.cluster.RedisClusterClient.connect(RedisClusterClient.java:717)
Error has been observed at the following site(s):
	*__Mono.fromCompletionStage ⇢ at io.lettuce.core.cluster.RedisClusterClient.connect(RedisClusterClient.java:717)
	|_           Mono.doOnError ⇢ at io.lettuce.core.cluster.RedisClusterClient.connect(RedisClusterClient.java:717)
	*________Mono.onErrorResume ⇢ at io.lettuce.core.cluster.RedisClusterClient.connectClusterAsync(RedisClusterClient.java:684)
	|_            Mono.doOnNext ⇢ at io.lettuce.core.cluster.RedisClusterClient.connectClusterAsync(RedisClusterClient.java:688)
	|_                 Mono.map ⇢ at io.lettuce.core.cluster.RedisClusterClient.connectClusterAsync(RedisClusterClient.java:690)
Original Stack Trace:
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261)
		at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
		at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)
		at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
		at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.base/java.lang.Thread.run(Thread.java:840)

Any ideas what might be missing here?

• I am on a Mac M1
• We have no problem connecting with the standard RedisClient to local standalone instances
• If we use the RedisClient to try to connect to a local cluster, we get a MOVED error
• The "Unable to connect" error only comes after switching to the RedisClusterClient
• I tried using "localhost" and the Docker service name (eg. "redis-cache-1") as the host in the URI
  • Also tried to use a single host versus the list of hosts

[ggivo]

On Mac docker runs in VM and is a bit tricky to get it right.
I assume the announced IP's by Redis cluster instances are from the docker bridge network inside the VM(172.25.0.2, 172.25.03, 172.25.0.4). They are not accessible from the host (MAC). If you are running your Java code on the MAC host directly it is not able to access those IP's. So when RedisClusterClient loads the cluster topology, it will try to contact above IP's and hence the error.

One way to overcome this is to set "--cluster-announce-ip 127.0.0.1" and "--cluster-announce-port " and make sure each node is mapped on different port

for more details you can check
https://redis.io/docs/latest/operate/oss_and_stack/management/config-file/

# In certain deployments, Redis Cluster nodes address discovery fails, because
# addresses are NAT-ted or because ports are forwarded (the typical case is
# Docker and other containers).
#
# In order to make Redis Cluster working in such environments, a static
# configuration where each node knows its public address is needed. The
# following four options are used for this scope, and are:
#
# * cluster-announce-ip
# * cluster-announce-port
# * cluster-announce-tls-port
# * cluster-announce-bus-port

[kioritanaka]

@ggivo Thank you for the quick turnaround!

This is what my compose file looks like after adding the two options you suggested, as well as making sure that there are no port conflicts.

Now, when I run the same create-cluster command:

docker exec -it redis-cache-1 redis-cli --cluster create redis-cache-1:6379 redis-cache2-1:6379 redis-cache3-1:6379 --cluster-replicas 0

the command hangs on "Waiting for the cluster to join."

I tried this same workflow to spin up a Redis instance in Ubuntu in an EC2 instance to test, and in that case the cluster is automatically created after running docker compose up -d (weirdly didn't have to run create-cluster), eg:

sudo docker exec -it ubuntu-cache-1 redis-cli
127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3
cluster_size:3
cluster_current_epoch:3
cluster_my_epoch:1
cluster_stats_messages_sent:0
cluster_stats_messages_received:0
total_cluster_links_buffer_limit_exceeded:0

I wasn't able to test connecting to the Ubuntu Redis instance through my app using the cluster aware client though.

Any other options/configurations I could be missing in the case of spinning the cluster up locally on my M1 with those options you suggested?

[ggivo]

I had a similar issue in my env but tottaly forgot about it.
If I remember correctly, it was related to a bug in redis-cli when using 127.0.0.1.

So, instead of loopback address 127.0.0.1, I changed the nodes to use the MAC host's actual IP (192....).
I also :
started Redis nodes on the same port exposed in docker container
exposed also the cluster management port

I ended up with something like
https://gist.github.com/ggivo/ce80b12ab75b40668010eb335103dacc
redis-cli --cluster create 192.168.86.43:6380 192.168.86.43:6381 192.168.86.43:6382 --cluster-replicas 0 --cluster-yes

By the way, if you need a way to start test or dev env on MAC, we have prepared a docker image that runs all Redis nodes in single container to avoid some of those issues. It is intended ONLY for Dev/Testing and is still in backing, so it can have some braking changes in the future.
To use it to start a 3 node cluster
docker run -p 3000-3002:3000-3002 --env REDIS_CLUSTER=yes redislabs/client-libs-test:8.0-M01

[tishun]

@kioritanaka did @ggivo suggestion help?

[kioritanaka]

Hi @ggivo @tishun,

Yes! I used the first suggestion of using my MAC's actual IP, starting the nodes on the same port exposed in the docker container, exposing the cluster management port, and making sure I had all the options in the example gist provided. Thank you so much!