connect ETIMEDOUT 172.17.0.6:7000

[sparachi]

I wrote an node api using thunk-redis, and testing from POSTman. The data get or set works sometimes and sometimes it just hangs in there, and times out with the error message "connect ETIMEDOUT 172.17.0.6:7000"

I followed steps here in https://github.com/Grokzen/docker-redis-cluster to create a docker image with 7000 -> 7002 exposed for clients.

The documentation is not so helpful w.r.t cluster configuration. How do I debug and isolate the issue?

[zensh]

Could you connect redis with redis-cli ?

[sparachi]

Yes I can connect to using cli. I can connect from the node app also, but the connection is not reliable. It hangs couple of times, and does not return anything.

[sparachi]

@zensh any debugging help would be appreciated.

[zensh]

@sparachi I think that this option killed the client. cluster-node-timeout 5000 https://github.com/Grokzen/docker-redis-cluster/blob/master/docker-data/redis-conf/7000/redis.conf#L4

[sparachi]

I have removed the entry in the redis.conf and restarted the docker container. But the issue still persists.

[zensh]

@sparachi I got the problem: redis internal IP and external IP are different.

when connect:

const cli = redis.createClient([
  "127.0.0.1:7000",
  "127.0.0.1:7001",
  "127.0.0.1:7002",
  "127.0.0.1:7003",
  "127.0.0.1:7004",
  "127.0.0.1:7005",
  "127.0.0.1:7006",
  "127.0.0.1:7007"
])

but cluster slots command give me:

[ [ 0, 5460, [ '172.17.0.2', 7000 ], [ '172.17.0.2', 7003 ] ],
  [ 10923, 16383, [ '172.17.0.2', 7002 ], [ '172.17.0.2', 7005 ] ],
  [ 5461, 10922, [ '172.17.0.2', 7001 ], [ '172.17.0.2', 7004 ] ] ]

that means:
If we send a command client.get("somekey") on connection 127.0.0.1:7000, redis cannot find the key on 7000, and respond a error (error) MOVED 9842 172.17.0.2:7001, then client try to connect to 172.17.0.2:7001, but it is internal IP!

You can try this with redis-cli, look like:

127.0.0.1:7000> get 1
(error) MOVED 9842 172.17.0.2:7001

[zensh]

I can resolve it with a new option IPMap

[sparachi]

@zensh I still the see the same behavior. Below is my modified createClient method.

I exposed all the 6 ports
docker run -d --name redis-cluster -p 7000:7000 -p 7001:7001 -p 7002:7002 -p 7003:7003 -p 7004:7004 -p 7005:7005 redis-cluster

(Ideally I think I need to do writes on master i.e. 7000 - 7002, reads on slaves 7003-7005. But for now in dev I exposed all the 6 ports since all are running on the same instance of docker container)

let clusterNodes = [
'127.0.0.1:7000',
'127.0.0.1:7001',
'127.0.0.1:7002',
'127.0.0.1:7003',
'127.0.0.1:7004',
'127.0.0.1:7005'
];

let options = {
clusterMode: true,
noDelay: true,
usePromise: blueBird,
IPMap: {
'172.17.0.7:7000': '127.0.0.1:7000',
'172.17.0.7:7001': '127.0.0.1:7001',
'172.17.0.7:7002': '127.0.0.1:7002',
'172.17.0.7:7003': '127.0.0.1:7003',
'172.17.0.7:7004': '127.0.0.1:7004',
'172.17.0.7:7005': '127.0.0.1:7005'
}
};

let redisClient = redis.createClient(clusterNodes, options);

In my GET method I do this

return new Promise(function(resolve, reject) {
// id is the parameter passed to this method
let data = {};
data.key = JSON.parse(id);
console.log("value to query " + data.key);
redisClient.get(data.key).then(function (res) {
console.log('data get from redis:', res);
if(res){
data.response = res;
} else {
data.response = 'null';
}
return redisClient.quit();
})
.then(function (res) {
console.log('redis client quit:', res);
// redis client quit: OK
return resolve(data);
})
.catch(function (err) {
console.error(err);
return reject(err);
});
});

In my SET method I do this below

return new Promise(function(resolve, reject) {
let data = {"status": "success"};

  redisClient.set(dataReceived.key, dataReceived.value)
  .then(function (res) {
    console.log('data set to redis: ', res);
    return redisClient.get(dataReceived.key);
  })
  .then(function (res) {
    console.log('data get from redis:', res);
    return redisClient.quit();
  })
  .then(function (res) {
    console.log('redis client quit:', res);
    // redis client quit: OK
    return resolve(data);
  })
  .catch(function (err) {
    console.error(err);
    return reject(err);
  });

});

Even though if it works, I have some questions around the production setup.

1. do I need to mention all the docker ip's, as well as EC2 host ip's when I createClient() in my node API. Since all my redis nodes dockerized, node api dockerized will be running in different EC2 instances and they will have dynamic IP's.
2. did someone try this in production deployment as you know of

[zensh]

@sparachi Currently no way to announce different IP/port. redis/redis#2704

[sparachi]

redis/redis#2527 (comment)
says it is implemented. I am confused here.

[zensh]

@sparachi

This proposal is implemented in Redis unstable branch. The first stable version of Redis to support that will be 4.0, released as RC1 15th of October 2016. Anyone testing this support with success in testing environments? Feedbacks appreciated.