All of a sudden, I'm receiving this error

[thelinuxlich]

[Error: Too many Cluster redirections. Last error: ReplyError: MOVED 6686 10.240.100.31:7000]

I have a 6 nodes setup, I saw this in the log of a node:

Oct 31 19:04:54 aidax-redis-2 redis-server[1501]: 1501:M 31 Oct 19:04:54.996 * Marking node bd1251b171bf467313c4300d761d92417cbb586f as failing (quorum reached).
Oct 31 19:04:55 aidax-redis-2 redis-server[1501]: 1501:M 31 Oct 19:04:55.439 * Clear FAIL state for node bd1251b171bf467313c4300d761d92417cbb586f: slave is reachable again.

I think ioredis is not recovering well alongside the redis cluster(3.2)

[luin]

Could you please enable the debug mode (DEBUG=ioredis:* node yourapp.js) and post the logs during this problem happens here?

[thelinuxlich]

Ok, I will keep you updated

[thelinuxlich]

So today it happened again and this is the log with DEBUG:

https://gist.github.com/thelinuxlich/365b8cafc98295b14077a38a44a949a7

[luin]

When a command is rejected with error MOVED, ioredis will update the local slot cache and send the command to the new node. It's strange that the command is sent multiple times to the same node.

[thelinuxlich]

I'm configuring it to connect with all the 6 nodes, and when this happens, I have to restart the whole process :(

[thelinuxlich]

Some more info: I'm using Node 5.5 and the redis operations consist of GET, SET, SETEX and EVALSHA scripts(using defineCommand)

[thelinuxlich]

Is there anymore info I can provide you to help pinpoint the issue?

[luin]

@thelinuxlich Is this issue related with your environment? If the issue cannot be reproduced with general setups, it would be very helpful to provide the docker files so I can debug the problem locally in exact the same environment as yours. Otherwise is it possible to help me narrow down the possibility where the problem happens.

[thelinuxlich]

I'm testing with older ioredis versions, let's see

[thelinuxlich]

Interesting, ioredis 2.0.0 is doing okay

[luin]

That's strange since there's not much difference between the two versions: v2.0.0...v2.4.0. Hmm...

[thelinuxlich]

Now I can confirm that switching to ioredis 2.0.0 totally mitigates the problem

[luin]

@thelinuxlich Could you please also test with ioredis v2.1.0 & v2.2.0 so we can know which version was the issue introduced. Thanks a lot 😆

[thelinuxlich]

After days of testing, can confirm this happens in ioredis 2.2+

[luin]

That means 2.1.0 doesn't have the problem? Looks like there's no much difference in cluster mode between 2.1.0 & 2.2.0: v2.1.0...v2.2.0. Hmm...

[AVVS]

To me it looks like cluster misconfiguration. the difference is that ioredis create a new node based on report from a cluster node, which tells to ask some new node not from the list - it tries to ask that node, gets another redirect and so on. One would assume that that nodes report wrong ip addresses for some reason and that leads to redirection loops

Other possibility is that response is parsed incorrectly? But I assume it's not the case though

[thelinuxlich]

Well, it can't be misconfiguration because I've been using ioredis for more than a year without changes in configuration and all of a sudden when I update from 2.0 to 2.4 this began to happen.

[AVVS]

@thelinuxlich maybe you are right. The place it can happen is

v2.1.0...v2.2.0#diff-ded17b5f6b1633a10701ad5e98a4e3c8R428

[Aditya-Chowdhry]

Recently used Ioredis cluster in production and got the above-mentioned errors. Just want to check up on the status of the issue. Is it resolved in latest version?

[luin]

@Aditya-Chowdhry No, the issue hasn't been reproduced in my environment.
@thelinuxlich How did you solve the problem in the end?
@AVVS The link you posted seems to be broken. Which part of code did you mention?

[AVVS]

https://github.com/luin/ioredis/compare/v2.1.0...v2.2.0#diff-ded17b5f6b1633a10701ad5e98a4e3c8R428

Fixed link

[luin]

@AVVS Thanks for the update. How is this line of code related with the issue?

[Aditya-Chowdhry]

In my case, I am only getting this same exact error repeatedly

MOVED 0 xxx.xx.xx.xx:6389

where the cluster consists of only 4 masters, ports - 6389,6390,6391,6392

I have another cluster consisting of master & slaves for which I am not getting any error.

[luin]

@Aditya-Chowdhry Was 6389 down while getting the error? Is the issue reproducible?

[Aditya-Chowdhry]

@luin No 6389 was not down. I am not able to reproduce it locally. Debugging further. Will get back if found something ASAP.

[Aditya-Chowdhry]

@luin Got the issue. This error that I posted comes when null cache key is passed. I wrote a rough small code to reproduce the issue.

'use strict';
const Redis = require('ioredis');
const CLUSTER_HOST = '127.0.0.1';


const cluster = new Redis.Cluster([{
    port: 6383,
    host: CLUSTER_HOST
}, {
    port: 6384,
    host: CLUSTER_HOST
}]);

let cacheKey = null;
let data = 'Hey this is test data';
console.log('CacheKey', cacheKey);
for(var i = 0; i < 100; ++i) {
	cluster.set(cacheKey, data, function(err, reply){
		if(err) console.log('Error', err);
		if(reply) console.log('Okay set done', reply);
		cluster.expire(cacheKey, 1234, function(err, reply){
			if(err) console.log('Error', err);
			if(reply) console.log('Okay expire done', reply);
			cluster.get(cacheKey, function(err, reply){
				if(err) console.log('Error', err);
				if(reply) console.log('Okay get done', reply);
			});
		});
	});
}

> Error: Too many Cluster redirections. Last error: ReplyError: MOVED 0 127.0.0.1:6380

[luin]

@Aditya-Chowdhry Just fixed this issue in v3.2.1. Thank you for the example, that helps!

[tuananh]

@luin one question: why when key undefined, you want to calculate slot anyway?

[luin]

@tuananh null and undefined are converted to empty strings in ioredis. It makes sense to calculate slot for an empty string.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[thenitai]

I'm seeing this issue with 3.2.2 in our production environment with 3 redis nodes (different IP's)