-
Notifications
You must be signed in to change notification settings - Fork 986
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow randomization of read candidates using Redis Cluster #834
Comments
Lettuce selects a slave per slot hash and uses the selected slave for read operations until the next topology refresh. This is by design to reduce CPU overhead. |
In high-concurrency, reading data from a slave node can improve request effectiveness if a master node in the cluster has multiple slave nodes and we can share read requests on these slave nodes. |
You can mitigate this behavior by enabling periodic topology refresh. Each time the topology is refreshed, Lettuce discards the slot hash connection cache and determines a different slave. This also works if there were no changes to the topology. |
How does lettuce do the refresh, by "cluster slot" command? if so, it also increase the host pressure and the "cluster slot" command spend a lot of time. I think it would be better to provide interface that we can DIY by ourselves. |
Sorry if I wasn't precise enough: Lettuce calculates the slot-hash itself and then determines read candidates for the hash slot by obtaining the node selection from |
in our project, the TPS is very high, we want to distribute read requests across slave nodes.
|
That's a neat approach to apply randomization without creating objects. Currently, what's missing is the ability to configure this behavior. We could introduce a |
I pushed a change to our feature branch for this ticket and the branch build that has produced artifact snapshots with version Make sure to include our snapshot repository: <dependencies>
<dependency>
<groupId>io.lettuce</groupId>
<artifactId>lettuce-core</artifactId>
<version>5.2.0.gh-834-SNAPSHOT</version>
</dependency>
</dependencies>
<repositories>
<repository>
<id>sonatype-snapshots</id>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories> |
if we go to the random branch, the connections would not be cached, does it affect performance? |
Connections are held in |
after testing, the read lantency maybe increases 5%. |
Thanks a lot for testing the changes. I reinstantiated caching of read candidates which restricts randomization to some extent. |
good job, my test result that the lantency reduces 4% one more thing, I find that the lettuce cluster would get the "cluster nodes" info and the "client list" info when refresh the topology refresh. I happened a problem that if a cluster is big which has hundreds of nodes and there are handreds of lettuce clients, the "cluster nodes" and "client list" info would be a large data and the commands would spend a lot of time, the redis instance's cpu rate would raise and the lantency would increase a lot. the default refreshPeriod is 60s, it is so short that in our situation the "CLUSTER" and "CLIENT" commands' tps becomes very high |
Lettuce uses the client count to determine the cluster node connect ordering (sort by least number of clients). This is, to connect to a default node. The default node is used for key-less commands (such as Pub/Sub). |
ReadFrom can now express whether it is order-sensitive or non-sensitive. This allows the using code to apply optimizations such as caching for order-sensitive implementation or randomization for non-sensitive implementations. ReadFrom.ANY is an implementation that allows reads from any node without considering the actual node ordering.
That change is in place now. |
@StillerRen How about increasing the refresh period and enabling adaptive timeouts (that's a different issue than this one, though). |
Just give advice, if some one uses the lettuce like this, he need pay attention to increasing the refresh period or set "false" for refresh. |
@mp911de As I see, the support for the read distribution from replicas will be added in the 5.2.x. When this going to be released? |
Probably September, see https://github.com/lettuce-io/lettuce-core/milestone/51. |
@mp911de Thanks for the quick reply. We are really waiting for this to be released. |
@mp911de @smoomrik AWS has recently introduced reader endpoints for Redis which manages distribution of reads among the slaves. You can check more details here: https://aws.amazon.com/about-aws/whats-new/2019/06/amazon-elasticache-launches-reader-endpoint-for-redis/ Might want to check this out. Using this should help avoid having to provide individual replica endpoints in the application giving more flexibility. |
hey @adityak93 , we have been using reader endpoint, but its not distributing the traffic equally among read replicas for us.sometimes, even when other read replicase are under heavy load, few read replicase have barely any requests coming their way. can you tell us if this is consistent with your observations , or if you used any additional configuration apart from read preferred and periodic topology refresh. |
Bug Report
Current Behavior
I am trying to connect to Redis cluster with 5 slave nodes and 1 master. All read queries are redirected to only one slave throughout the application scope.
Input Code
Expected behavior/code
It should distribute read queries on all slaves.
Environment
Possible Solution
Additional context
The text was updated successfully, but these errors were encountered: