Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lettuce 4.4.4 Final version During failover the client make a lot of connections to redis #944

Closed
s-aravind-flipkart opened this issue Dec 17, 2018 · 7 comments
Labels
status: waiting-for-feedback We need additional information before we can continue status: waiting-for-triage

Comments

@s-aravind-flipkart
Copy link

Redis cluster went into limbo state when the client started to make a lot of connections to redis cluster. I've not enabled any pooling as such. During the failover state the clients were making 1000 connections to 78 boxes.

I think definitely there is a connection leak in the client. loadViews() was called from the client. Redis cluster was not able to handle so many connections and was constantly rebalancing between master to slave, slave to master as such.

How do we avoid this condition in future.

@s-aravind-flipkart
Copy link
Author

xxx warning redis[xxxx]: Failover auth denied to 2e07xxx: already voted for epoch 225

@mp911de mp911de added status: waiting-for-feedback We need additional information before we can continue status: waiting-for-triage labels Dec 17, 2018
@mp911de
Copy link
Collaborator

mp911de commented Dec 17, 2018

Potentially related tickets: #342, #721.

You can force a connection leak by imposing a lower command execution timeout than the connect timeout (e.g. RedisURI timeout is 10 seconds, timeout set via RedisClusterClient.setDefaultTimeout(…) is higher than 10 seconds). Can you check your timeout settings?

@s-aravind-flipkart
Copy link
Author

@mp911de The connection time out is 2x the request timeout. Im trying deploying the config changes to see the effect

@s-aravind-flipkart
Copy link
Author

@mp911de Im using the future.get() .get timeout was not same as connection time out.

@s-aravind-flipkart
Copy link
Author

The default timeout was 5s in cluster client. In Redis future from lettuce, we used the timeout to 1s.

@mp911de
Copy link
Collaborator

mp911de commented Jan 2, 2019

Did you change timeouts and is the issue now solved?

@mp911de
Copy link
Collaborator

mp911de commented Jan 8, 2019

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

@mp911de mp911de closed this as completed Jan 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: waiting-for-feedback We need additional information before we can continue status: waiting-for-triage
Projects
None yet
Development

No branches or pull requests

2 participants