Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cluster mode connections growing over time #1449

Closed
Cellule opened this issue Oct 25, 2021 · 10 comments
Closed

Cluster mode connections growing over time #1449

Cellule opened this issue Oct 25, 2021 · 10 comments

Comments

@Cellule
Copy link

Cellule commented Oct 25, 2021

We've recently been seeing huge slowdowns on our servers. We can't totally blame on redis yet, but we've noticed a huge amount of connections on our redis instance
It seems to go down only when we restart the servers.
For context, we have roughly 16 servers running all creating 4 connections each to the cluster.
The connections are long-lived (aka we never create new ones during the process). So I can only assume something in the reconnection behavior maybe isn't cleaning up the previous connections.

We are currently using ioredis version 4.19.2. I'll look at updating to latest see if it helps, but I haven't seen anything in the changelog that would suggest a fix. I also stumbled on #1295 which was closed for lack of more info.
I can provide as much info as I can. Not sure I can run DEBUG=ioredis:* on our production servers, but here's what we have from cloudwatch for the past 3 months where we sometimes peak over 700 connections

image

I also ran client list on our redis instance a few hours after restarting many of our servers if it can help. We can see that a single addr can have multiple connections to the same topic(name)

id=65720874 addr=10.18.17.153:58354 fd=29 name=publisher age=12373 idle=12373 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65720875 addr=10.18.17.153:58360 fd=30 name=subscriber age=12373 idle=12373 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716136 addr=10.18.37.92:54730 fd=91 name=locks age=13348 idle=13348 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716137 addr=10.18.37.92:54734 fd=92 name=subscriber age=13348 idle=13348 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716138 addr=10.18.37.92:54746 fd=93 name=cache age=13348 idle=13348 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536130 addr=10.18.96.176:33002 fd=452 name=publisher age=243524 idle=243524 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760195 addr=10.18.8.64:43558 fd=39 name=locks age=4282 idle=4282 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65790643 addr=10.18.41.124:53862 fd=154 name=cache age=1 idle=1 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65790644 addr=10.18.41.124:53882 fd=160 name=locks age=1 idle=1 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65790645 addr=10.18.41.124:53876 fd=161 name=subscriber age=1 idle=1 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65790646 addr=10.18.41.124:53870 fd=162 name=publisher age=1 idle=1 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760196 addr=10.18.8.64:43564 fd=46 name=subscriber age=4282 idle=4282 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760197 addr=10.18.8.64:43568 fd=48 name=publisher age=4282 idle=4282 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772282 addr=10.18.44.236:40478 fd=118 name=cache age=2385 idle=2385 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772283 addr=10.18.44.236:40482 fd=119 name=subscriber age=2385 idle=2385 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772284 addr=10.18.44.236:40490 fd=120 name=locks age=2385 idle=2385 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777144 addr=10.18.22.131:47932 fd=28 name=locks age=1685 idle=1685 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777145 addr=10.18.22.131:47940 fd=127 name=cache age=1685 idle=1685 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777146 addr=10.18.22.131:47946 fd=129 name=subscriber age=1685 idle=1685 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777147 addr=10.18.22.131:47952 fd=130 name=publisher age=1685 idle=1685 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772285 addr=10.18.44.236:40496 fd=121 name=publisher age=2385 idle=2385 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760198 addr=10.18.8.64:43576 fd=49 name=cache age=4282 idle=4282 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536131 addr=10.18.96.176:33006 fd=453 name=subscriber age=243524 idle=243524 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536132 addr=10.18.96.176:33014 fd=454 name=locks age=243524 idle=243524 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687647 addr=10.18.8.64:44646 fd=640 name=cache age=19295 idle=19295 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65717700 addr=10.18.4.139:43520 fd=20 name=ioredisClusterSubscriber age=13041 idle=2 flags=P db=0 sub=66 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe user=default
id=65776062 addr=10.18.22.131:46872 fd=113 name=cache age=1832 idle=1832 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65776063 addr=10.18.22.131:46878 fd=124 name=publisher age=1832 idle=1832 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65776064 addr=10.18.22.131:46890 fd=125 name=subscriber age=1832 idle=1832 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65776065 addr=10.18.22.131:46884 fd=126 name=locks age=1832 idle=1832 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687648 addr=10.18.8.64:44650 fd=641 name=locks age=19295 idle=19295 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687649 addr=10.18.8.64:44656 fd=642 name=publisher age=19295 idle=19295 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760779 addr=10.18.8.64:44420 fd=55 name=cache age=4182 idle=4182 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760780 addr=10.18.8.64:44426 fd=76 name=subscriber age=4182 idle=4182 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760781 addr=10.18.8.64:44432 fd=107 name=locks age=4182 idle=4182 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760782 addr=10.18.8.64:44438 fd=108 name=publisher age=4182 idle=4182 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687650 addr=10.18.8.64:44662 fd=643 name=subscriber age=19295 idle=19295 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716078 addr=10.18.45.59:35620 fd=71 name=publisher age=13357 idle=13357 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716079 addr=10.18.45.59:35628 fd=75 name=cache age=13357 idle=13357 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716080 addr=10.18.45.59:35634 fd=77 name=subscriber age=13357 idle=13357 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716081 addr=10.18.45.59:35640 fd=78 name=locks age=13357 idle=13357 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688488 addr=10.18.22.131:42866 fd=669 name=cache age=19269 idle=19269 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702675 addr=10.18.4.128:52144 fd=63 name=locks age=16274 idle=16274 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702676 addr=10.18.4.128:52150 fd=72 name=subscriber age=16274 idle=16274 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702677 addr=10.18.4.128:52162 fd=73 name=publisher age=16274 idle=16274 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65775854 addr=10.18.17.153:59018 fd=50 name=cache age=1862 idle=1862 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65775855 addr=10.18.17.153:59024 fd=62 name=locks age=1862 idle=1862 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65775856 addr=10.18.17.153:59030 fd=112 name=publisher age=1862 idle=1862 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65775857 addr=10.18.17.153:59036 fd=122 name=subscriber age=1862 idle=1862 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702678 addr=10.18.4.128:52156 fd=74 name=cache age=16274 idle=16274 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688490 addr=10.18.22.131:42870 fd=668 name=locks age=19269 idle=19269 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716125 addr=10.18.17.56:39010 fd=87 name=publisher age=13350 idle=13350 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716126 addr=10.18.17.56:39004 fd=88 name=cache age=13350 idle=13350 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716127 addr=10.18.17.56:39016 fd=89 name=locks age=13350 idle=13350 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716128 addr=10.18.17.56:39020 fd=23 name=subscriber age=13350 idle=13350 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688491 addr=10.18.22.131:42876 fd=670 name=publisher age=19269 idle=19269 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781048 addr=10.18.4.128:40060 fd=137 name=locks age=1178 idle=1178 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781049 addr=10.18.4.128:40068 fd=138 name=cache age=1178 idle=1178 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781050 addr=10.18.4.128:40074 fd=139 name=subscriber age=1178 idle=1178 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781051 addr=10.18.4.128:40080 fd=140 name=publisher age=1178 idle=1178 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755324 addr=10.18.24.157:36778 fd=24 name=cache age=5152 idle=5151 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755325 addr=10.18.24.157:36786 fd=40 name=subscriber age=5152 idle=5151 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755326 addr=10.18.24.157:36790 fd=38 name=locks age=5152 idle=5151 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755327 addr=10.18.24.157:36798 fd=41 name=publisher age=5152 idle=5151 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65698935 addr=10.18.4.128:48128 fd=65 name=locks age=17163 idle=17163 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65698937 addr=10.18.4.128:48134 fd=66 name=publisher age=17163 idle=17162 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688493 addr=10.18.22.131:42882 fd=675 name=subscriber age=19269 idle=19269 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65689122 addr=10.18.33.201:60198 fd=629 name=ioredisClusterSubscriber age=19249 idle=0 flags=P db=0 sub=16527 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe user=default
id=65754199 addr=10.18.44.236:46660 fd=32 name=cache age=5373 idle=5373 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65754200 addr=10.18.44.236:46666 fd=33 name=locks age=5373 idle=5373 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65754201 addr=10.18.44.236:46672 fd=34 name=publisher age=5373 idle=5373 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65754202 addr=10.18.44.236:46680 fd=35 name=subscriber age=5373 idle=5373 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688028 addr=10.18.4.128:39884 fd=654 name=subscriber age=19283 idle=19283 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716162 addr=10.18.10.53:51644 fd=94 name=cache age=13344 idle=13344 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716163 addr=10.18.10.53:51648 fd=95 name=publisher age=13344 idle=13344 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788713 addr=10.18.41.124:52478 fd=155 name=cache age=219 idle=219 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788714 addr=10.18.41.124:52488 fd=156 name=locks age=219 idle=219 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788715 addr=10.18.41.124:52482 fd=157 name=publisher age=219 idle=219 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788716 addr=10.18.41.124:52498 fd=158 name=subscriber age=219 idle=219 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716485 addr=10.18.30.184:52726 fd=102 name=ioredisClusterSubscriber age=13292 idle=24 flags=P db=0 sub=56 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe user=default
id=65716164 addr=10.18.10.53:51656 fd=96 name=locks age=13344 idle=13344 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716165 addr=10.18.10.53:51662 fd=97 name=subscriber age=13344 idle=13344 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65689332 addr=10.18.44.236:54302 fd=679 name=ioredisClusterSubscriber age=19243 idle=0 flags=P db=0 sub=16419 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=subscribe user=default
id=65688024 addr=10.18.4.128:39866 fd=648 name=locks age=19283 idle=19283 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688025 addr=10.18.4.128:39872 fd=652 name=cache age=19283 idle=19283 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688026 addr=10.18.4.128:39878 fd=653 name=publisher age=19283 idle=19283 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536112 addr=10.18.96.90:52348 fd=450 name=cache age=243524 idle=243524 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781007 addr=10.18.22.131:52882 fd=143 name=locks age=1184 idle=1184 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781008 addr=10.18.22.131:52896 fd=144 name=subscriber age=1184 idle=1184 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781009 addr=10.18.22.131:52888 fd=145 name=publisher age=1184 idle=1184 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688110 addr=10.18.44.236:54190 fd=655 name=cache age=19281 idle=19281 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716098 addr=10.18.30.184:52574 fd=83 name=locks age=13355 idle=13355 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716099 addr=10.18.30.184:52580 fd=84 name=cache age=13355 idle=13355 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716100 addr=10.18.30.184:52588 fd=85 name=publisher age=13355 idle=13355 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716101 addr=10.18.30.184:52594 fd=86 name=subscriber age=13355 idle=13355 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760407 addr=10.18.44.236:53750 fd=47 name=locks age=4244 idle=4244 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760408 addr=10.18.44.236:53756 fd=51 name=publisher age=4244 idle=4244 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760409 addr=10.18.44.236:53760 fd=52 name=subscriber age=4244 idle=4244 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760410 addr=10.18.44.236:53768 fd=53 name=cache age=4244 idle=4244 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688111 addr=10.18.44.236:54194 fd=661 name=locks age=19281 idle=19281 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688112 addr=10.18.44.236:54200 fd=662 name=publisher age=19281 idle=19281 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688113 addr=10.18.44.236:54206 fd=663 name=subscriber age=19281 idle=19281 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687864 addr=10.18.17.153:59160 fd=644 name=cache age=19288 idle=19288 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687865 addr=10.18.17.153:59168 fd=645 name=subscriber age=19288 idle=19288 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687866 addr=10.18.17.153:59174 fd=646 name=locks age=19288 idle=19288 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687867 addr=10.18.17.153:59180 fd=647 name=publisher age=19288 idle=19288 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716193 addr=10.18.4.139:43006 fd=98 name=publisher age=13340 idle=13340 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716194 addr=10.18.4.139:42994 fd=99 name=cache age=13340 idle=13340 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716195 addr=10.18.4.139:42986 fd=100 name=locks age=13340 idle=13340 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772237 addr=10.18.22.131:43732 fd=114 name=locks age=2392 idle=2392 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772238 addr=10.18.22.131:43726 fd=115 name=cache age=2392 idle=2392 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772239 addr=10.18.22.131:43738 fd=116 name=subscriber age=2392 idle=2392 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788127 addr=10.18.4.128:47118 fd=149 name=locks age=289 idle=289 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772240 addr=10.18.22.131:43746 fd=117 name=publisher age=2392 idle=2392 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716196 addr=10.18.4.139:43000 fd=101 name=subscriber age=13340 idle=13340 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65698933 addr=10.18.4.128:48118 fd=56 name=cache age=17163 idle=17163 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65759603 addr=10.18.8.64:42794 fd=42 name=locks age=4386 idle=4386 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781003 addr=10.18.22.131:52876 fd=136 name=cache age=1184 idle=1184 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65759604 addr=10.18.8.64:42790 fd=43 name=subscriber age=4386 idle=4386 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65759605 addr=10.18.8.64:42808 fd=44 name=cache age=4386 idle=4386 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65759606 addr=10.18.8.64:42802 fd=45 name=publisher age=4386 idle=4386 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65698934 addr=10.18.4.128:48122 fd=64 name=subscriber age=17163 idle=17163 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688106 addr=10.18.41.124:41790 fd=658 name=publisher age=19281 idle=19281 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688107 addr=10.18.41.124:41794 fd=659 name=locks age=19281 idle=19281 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688108 addr=10.18.41.124:41802 fd=660 name=subscriber age=19281 idle=19281 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536109 addr=10.18.96.90:52330 fd=447 name=subscriber age=243524 idle=243524 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65692871 addr=10.18.8.64:47314 fd=57 name=publisher age=18682 idle=18682 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788124 addr=10.18.4.128:47098 fd=146 name=cache age=289 idle=289 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788125 addr=10.18.4.128:47104 fd=147 name=publisher age=289 idle=289 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788126 addr=10.18.4.128:47108 fd=148 name=subscriber age=289 idle=289 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65692872 addr=10.18.8.64:47320 fd=59 name=cache age=18682 idle=18682 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65692873 addr=10.18.8.64:47324 fd=60 name=locks age=18682 idle=18682 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65692874 addr=10.18.8.64:47332 fd=61 name=subscriber age=18682 idle=18682 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777194 addr=10.18.8.64:45570 fd=54 name=subscriber age=1677 idle=1677 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777195 addr=10.18.8.64:45578 fd=131 name=cache age=1677 idle=1677 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777196 addr=10.18.8.64:45590 fd=133 name=locks age=1677 idle=1677 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777197 addr=10.18.8.64:45584 fd=134 name=publisher age=1677 idle=1677 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702225 addr=10.18.41.124:53200 fd=67 name=locks age=16376 idle=16376 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702226 addr=10.18.41.124:53208 fd=68 name=publisher age=16376 idle=16376 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702227 addr=10.18.41.124:53214 fd=69 name=cache age=16376 idle=16376 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702228 addr=10.18.41.124:53220 fd=70 name=subscriber age=16376 idle=16376 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536110 addr=10.18.96.90:52334 fd=448 name=publisher age=243524 idle=243524 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536111 addr=10.18.96.90:52338 fd=449 name=locks age=243524 idle=243524 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=18951707 addr=10.18.83.50:39236 fd=151 name=locks age=4753462 idle=4753461 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716091 addr=10.18.2.236:38748 fd=79 name=cache age=13355 idle=13355 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716092 addr=10.18.2.236:38754 fd=80 name=subscriber age=13355 idle=13355 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716093 addr=10.18.2.236:38768 fd=81 name=locks age=13355 idle=13355 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716094 addr=10.18.2.236:38762 fd=82 name=publisher age=13355 idle=13355 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688098 addr=10.18.41.124:41782 fd=656 name=cache age=19281 idle=19281 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=18951708 addr=10.18.83.50:39242 fd=25 name=publisher age=4753461 idle=4753461 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=18951705 addr=10.18.83.50:39228 fd=58 name=subscriber age=4753462 idle=4753462 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=18951706 addr=10.18.83.50:39224 fd=150 name=cache age=4753462 idle=4753462 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688254 addr=10.18.33.201:60056 fd=665 name=cache age=19276 idle=19276 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688257 addr=10.18.33.201:60064 fd=664 name=locks age=19276 idle=19276 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688258 addr=10.18.33.201:60070 fd=666 name=subscriber age=19276 idle=19276 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688259 addr=10.18.33.201:60074 fd=667 name=publisher age=19276 idle=19276 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687836 addr=10.18.24.157:36980 fd=638 name=subscriber age=19289 idle=19289 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687837 addr=10.18.24.157:36974 fd=649 name=publisher age=19289 idle=19289 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716247 addr=10.18.34.93:55534 fd=103 name=subscriber age=13330 idle=13330 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716248 addr=10.18.34.93:55530 fd=104 name=cache age=13330 idle=13330 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716249 addr=10.18.34.93:55540 fd=105 name=publisher age=13330 idle=13330 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65762843 addr=10.18.8.64:47768 fd=21 name=cache age=3849 idle=3849 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65762844 addr=10.18.8.64:47776 fd=109 name=locks age=3849 idle=3849 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65762845 addr=10.18.8.64:47788 fd=110 name=subscriber age=3849 idle=3849 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65762846 addr=10.18.8.64:47782 fd=111 name=publisher age=3849 idle=3849 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716250 addr=10.18.34.93:55546 fd=106 name=locks age=13330 idle=13330 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755042 addr=10.18.44.236:47584 fd=22 name=publisher age=5205 idle=5205 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755043 addr=10.18.44.236:47590 fd=31 name=subscriber age=5205 idle=5205 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755044 addr=10.18.44.236:47594 fd=36 name=locks age=5205 idle=5205 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788555 addr=10.18.24.157:34954 fd=123 name=locks age=237 idle=237 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788556 addr=10.18.24.157:34960 fd=141 name=publisher age=237 idle=237 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788557 addr=10.18.24.157:34966 fd=152 name=subscriber age=237 idle=237 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788558 addr=10.18.24.157:34972 fd=153 name=cache age=237 idle=237 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755045 addr=10.18.44.236:47576 fd=37 name=cache age=5205 idle=5205 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687838 addr=10.18.24.157:36968 fd=650 name=locks age=19289 idle=19289 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687839 addr=10.18.24.157:36964 fd=651 name=cache age=19289 idle=19289 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536129 addr=10.18.96.176:32996 fd=451 name=cache age=243524 idle=243524 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65716135 addr=10.18.37.92:54740 fd=90 name=publisher age=13348 idle=13348 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65720871 addr=10.18.17.153:58340 fd=26 name=locks age=12373 idle=12373 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65720872 addr=10.18.17.153:58348 fd=27 name=cache age=12373 idle=12373 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
@Cellule
Copy link
Author

Cellule commented Oct 26, 2021

To give an idea, since I created this issue yesterday (about 17hours ago) the number of connections went from ~180 to ~280

Might be useful to know that out of our 16 servers, 8 are for production and 8 are for beta. Whenever we redeploy to the beta servers, the connections barely changes, which seems to hint that our production servers are using pretty much all the connections. So I'm guessing that work load probably have an impact

id=66441957 addr=10.18.39.102:56030 fd=276 name=cache age=3933 idle=3933 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66426008 addr=10.18.44.236:44326 fd=245 name=cache age=5121 idle=5121 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65880316 addr=10.18.33.201:55070 fd=199 name=locks age=52864 idle=52864 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65880317 addr=10.18.33.201:55082 fd=201 name=subscriber age=52864 idle=52864 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65880318 addr=10.18.33.201:55076 fd=202 name=publisher age=52864 idle=52864 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65790646 addr=10.18.41.124:53870 fd=162 name=publisher age=61866 idle=61866 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760196 addr=10.18.8.64:43564 fd=46 name=subscriber age=66147 idle=66147 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760197 addr=10.18.8.64:43568 fd=48 name=publisher age=66147 idle=66147 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772282 addr=10.18.44.236:40478 fd=118 name=cache age=64250 idle=64250 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772283 addr=10.18.44.236:40482 fd=119 name=subscriber age=64250 idle=64250 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772284 addr=10.18.44.236:40490 fd=120 name=locks age=64250 idle=64250 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777144 addr=10.18.22.131:47932 fd=28 name=locks age=63550 idle=63550 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777145 addr=10.18.22.131:47940 fd=127 name=cache age=63550 idle=63550 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777146 addr=10.18.22.131:47946 fd=129 name=subscriber age=63550 idle=63550 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777147 addr=10.18.22.131:47952 fd=130 name=publisher age=63550 idle=63550 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772285 addr=10.18.44.236:40496 fd=121 name=publisher age=64250 idle=64250 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760198 addr=10.18.8.64:43576 fd=49 name=cache age=66147 idle=66147 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65832594 addr=10.18.17.153:47090 fd=175 name=cache age=57403 idle=57403 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65832595 addr=10.18.17.153:47096 fd=181 name=publisher age=57403 idle=57403 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441799 addr=10.18.44.29:51442 fd=241 name=publisher age=3942 idle=3942 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441800 addr=10.18.44.29:51446 fd=252 name=locks age=3942 idle=3942 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441801 addr=10.18.44.29:51452 fd=263 name=subscriber age=3942 idle=3942 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441802 addr=10.18.44.29:51460 fd=264 name=cache age=3942 idle=3942 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65832596 addr=10.18.17.153:47102 fd=182 name=subscriber age=57403 idle=57403 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65832597 addr=10.18.17.153:47110 fd=183 name=locks age=57403 idle=57403 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536131 addr=10.18.96.176:33006 fd=453 name=subscriber age=305389 idle=305389 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536132 addr=10.18.96.176:33014 fd=454 name=locks age=305389 idle=305389 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65883960 addr=10.18.22.131:38986 fd=191 name=locks age=52527 idle=52527 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65883961 addr=10.18.22.131:38992 fd=197 name=cache age=52527 idle=52527 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65883962 addr=10.18.22.131:39000 fd=203 name=subscriber age=52527 idle=52527 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65883963 addr=10.18.22.131:39006 fd=204 name=publisher age=52527 idle=52527 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687647 addr=10.18.8.64:44646 fd=640 name=cache age=81160 idle=81160 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65798125 addr=10.18.41.124:60618 fd=165 name=cache age=61012 idle=61012 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65798126 addr=10.18.41.124:60624 fd=167 name=locks age=61012 idle=61012 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65798127 addr=10.18.41.124:60630 fd=168 name=subscriber age=61012 idle=61012 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66368102 addr=10.18.44.236:60198 fd=217 name=publisher age=9808 idle=9808 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66368103 addr=10.18.44.236:60204 fd=222 name=locks age=9808 idle=9808 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66368104 addr=10.18.44.236:60210 fd=223 name=cache age=9808 idle=9808 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66368105 addr=10.18.44.236:60216 fd=224 name=subscriber age=9808 idle=9808 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65798128 addr=10.18.41.124:60634 fd=169 name=publisher age=61012 idle=61012 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65776062 addr=10.18.22.131:46872 fd=113 name=cache age=63697 idle=63697 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65948360 addr=10.18.22.131:45176 fd=198 name=locks age=46749 idle=46749 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65948361 addr=10.18.22.131:45182 fd=210 name=subscriber age=46749 idle=46749 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65948362 addr=10.18.22.131:45188 fd=211 name=publisher age=46749 idle=46749 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65948363 addr=10.18.22.131:45194 fd=212 name=cache age=46749 idle=46749 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66398639 addr=10.18.44.236:36456 fd=213 name=cache age=7303 idle=7303 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66398640 addr=10.18.44.236:36462 fd=226 name=publisher age=7303 idle=7303 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66398641 addr=10.18.44.236:36468 fd=231 name=locks age=7303 idle=7303 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66398642 addr=10.18.44.236:36474 fd=232 name=subscriber age=7303 idle=7303 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65776063 addr=10.18.22.131:46878 fd=124 name=publisher age=63697 idle=63697 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65776064 addr=10.18.22.131:46890 fd=125 name=subscriber age=63697 idle=63697 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65776065 addr=10.18.22.131:46884 fd=126 name=locks age=63697 idle=63697 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687648 addr=10.18.8.64:44650 fd=641 name=locks age=81160 idle=81160 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687649 addr=10.18.8.64:44656 fd=642 name=publisher age=81160 idle=81160 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65819241 addr=10.18.8.64:54954 fd=176 name=locks age=58773 idle=58773 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65819242 addr=10.18.8.64:54960 fd=177 name=cache age=58773 idle=58773 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65837200 addr=10.18.24.157:49908 fd=193 name=publisher age=56952 idle=56952 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65837201 addr=10.18.24.157:49900 fd=194 name=cache age=56952 idle=56952 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65837202 addr=10.18.24.157:49912 fd=195 name=locks age=56952 idle=56952 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65837203 addr=10.18.24.157:49920 fd=196 name=subscriber age=56952 idle=56952 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65819243 addr=10.18.8.64:54966 fd=178 name=publisher age=58773 idle=58773 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65819244 addr=10.18.8.64:54972 fd=179 name=subscriber age=58773 idle=58773 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760779 addr=10.18.8.64:44420 fd=55 name=cache age=66047 idle=66047 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760780 addr=10.18.8.64:44426 fd=76 name=subscriber age=66047 idle=66047 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760781 addr=10.18.8.64:44432 fd=107 name=locks age=66047 idle=66047 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760782 addr=10.18.8.64:44438 fd=108 name=publisher age=66047 idle=66047 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687650 addr=10.18.8.64:44662 fd=643 name=subscriber age=81160 idle=81160 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441782 addr=10.18.6.34:36948 fd=253 name=subscriber age=3944 idle=3943 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441783 addr=10.18.6.34:36956 fd=255 name=cache age=3944 idle=3943 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441784 addr=10.18.6.34:36962 fd=256 name=locks age=3944 idle=3943 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441785 addr=10.18.6.34:36968 fd=258 name=publisher age=3944 idle=3943 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688488 addr=10.18.22.131:42866 fd=669 name=cache age=81134 idle=81134 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702675 addr=10.18.4.128:52144 fd=63 name=locks age=78139 idle=78139 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702676 addr=10.18.4.128:52150 fd=72 name=subscriber age=78139 idle=78139 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702677 addr=10.18.4.128:52162 fd=73 name=publisher age=78139 idle=78139 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65775854 addr=10.18.17.153:59018 fd=50 name=cache age=63727 idle=63727 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65775855 addr=10.18.17.153:59024 fd=62 name=locks age=63727 idle=63727 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65775856 addr=10.18.17.153:59030 fd=112 name=publisher age=63727 idle=63727 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65775857 addr=10.18.17.153:59036 fd=122 name=subscriber age=63727 idle=63727 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702678 addr=10.18.4.128:52156 fd=74 name=cache age=78139 idle=78139 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66442118 addr=10.18.39.61:55412 fd=281 name=subscriber age=3923 idle=3923 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66442119 addr=10.18.39.61:55418 fd=282 name=cache age=3923 idle=3923 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66442120 addr=10.18.39.61:55424 fd=283 name=publisher age=3923 idle=3923 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688490 addr=10.18.22.131:42870 fd=668 name=locks age=81134 idle=81134 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688491 addr=10.18.22.131:42876 fd=670 name=publisher age=81134 idle=81134 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781048 addr=10.18.4.128:40060 fd=137 name=locks age=63043 idle=63043 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781049 addr=10.18.4.128:40068 fd=138 name=cache age=63043 idle=63043 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781050 addr=10.18.4.128:40074 fd=139 name=subscriber age=63043 idle=63043 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66423849 addr=10.18.44.236:41330 fd=225 name=locks age=5293 idle=5293 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66423850 addr=10.18.44.236:41338 fd=237 name=cache age=5293 idle=5293 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66423851 addr=10.18.44.236:41344 fd=238 name=publisher age=5293 idle=5293 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66423852 addr=10.18.44.236:41350 fd=240 name=subscriber age=5293 idle=5293 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781051 addr=10.18.4.128:40080 fd=140 name=publisher age=63043 idle=63043 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755324 addr=10.18.24.157:36778 fd=24 name=cache age=67017 idle=67016 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755325 addr=10.18.24.157:36786 fd=40 name=subscriber age=67017 idle=67016 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755326 addr=10.18.24.157:36790 fd=38 name=locks age=67017 idle=67016 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755327 addr=10.18.24.157:36798 fd=41 name=publisher age=67017 idle=67016 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65698935 addr=10.18.4.128:48128 fd=65 name=locks age=79028 idle=79028 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65698937 addr=10.18.4.128:48134 fd=66 name=publisher age=79028 idle=79027 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688493 addr=10.18.22.131:42882 fd=675 name=subscriber age=81134 idle=81134 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65689122 addr=10.18.33.201:60198 fd=629 name=ioredisClusterSubscriber age=81114 idle=0 flags=P db=0 sub=34890 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe user=default
id=66428506 addr=10.18.17.153:59782 fd=239 name=subscriber age=4933 idle=4933 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66428507 addr=10.18.17.153:59788 fd=249 name=cache age=4933 idle=4933 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66428508 addr=10.18.17.153:59794 fd=250 name=publisher age=4933 idle=4933 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66442117 addr=10.18.39.61:55408 fd=284 name=locks age=3923 idle=3923 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66428509 addr=10.18.17.153:59800 fd=251 name=locks age=4933 idle=4933 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65754199 addr=10.18.44.236:46660 fd=32 name=cache age=67238 idle=67238 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65754200 addr=10.18.44.236:46666 fd=33 name=locks age=67238 idle=67238 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65754201 addr=10.18.44.236:46672 fd=34 name=publisher age=67238 idle=67238 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65754202 addr=10.18.44.236:46680 fd=35 name=subscriber age=67238 idle=67238 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688028 addr=10.18.4.128:39884 fd=654 name=subscriber age=81148 idle=81148 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788713 addr=10.18.41.124:52478 fd=155 name=cache age=62084 idle=62084 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788714 addr=10.18.41.124:52488 fd=156 name=locks age=62084 idle=62084 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66469328 addr=10.18.24.157:52214 fd=20 name=subscriber age=1884 idle=1884 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66469329 addr=10.18.24.157:52220 fd=71 name=locks age=1884 idle=1884 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66469330 addr=10.18.24.157:52232 fd=84 name=cache age=1884 idle=1884 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66469331 addr=10.18.24.157:52226 fd=86 name=publisher age=1884 idle=1884 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788715 addr=10.18.41.124:52482 fd=157 name=publisher age=62084 idle=62084 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788716 addr=10.18.41.124:52498 fd=158 name=subscriber age=62084 idle=62084 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65689332 addr=10.18.44.236:54302 fd=679 name=ioredisClusterSubscriber age=81108 idle=0 flags=P db=0 sub=35775 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=subscribe user=default
id=65688024 addr=10.18.4.128:39866 fd=648 name=locks age=81148 idle=81148 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688025 addr=10.18.4.128:39872 fd=652 name=cache age=81148 idle=81148 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688026 addr=10.18.4.128:39878 fd=653 name=publisher age=81148 idle=81148 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536112 addr=10.18.96.90:52348 fd=450 name=cache age=305389 idle=305389 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781007 addr=10.18.22.131:52882 fd=143 name=locks age=63049 idle=63049 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781008 addr=10.18.22.131:52896 fd=144 name=subscriber age=63049 idle=63049 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781009 addr=10.18.22.131:52888 fd=145 name=publisher age=63049 idle=63049 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688110 addr=10.18.44.236:54190 fd=655 name=cache age=81146 idle=81146 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65798630 addr=10.18.24.157:43276 fd=163 name=cache age=60955 idle=60955 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65798631 addr=10.18.24.157:43280 fd=170 name=publisher age=60955 idle=60955 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65798632 addr=10.18.24.157:43294 fd=171 name=subscriber age=60955 idle=60955 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65798633 addr=10.18.24.157:43288 fd=172 name=locks age=60955 idle=60955 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760407 addr=10.18.44.236:53750 fd=47 name=locks age=66109 idle=66109 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760408 addr=10.18.44.236:53756 fd=51 name=publisher age=66109 idle=66109 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760409 addr=10.18.44.236:53760 fd=52 name=subscriber age=66109 idle=66109 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65797698 addr=10.18.44.236:44000 fd=132 name=cache age=61060 idle=61060 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65797699 addr=10.18.44.236:43996 fd=142 name=locks age=61060 idle=61060 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65834483 addr=10.18.4.128:38504 fd=186 name=locks age=57215 idle=57215 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65834484 addr=10.18.4.128:38508 fd=180 name=cache age=57215 idle=57215 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65834485 addr=10.18.4.128:38516 fd=187 name=publisher age=57215 idle=57215 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65834486 addr=10.18.4.128:38522 fd=188 name=subscriber age=57215 idle=57215 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65797700 addr=10.18.44.236:44008 fd=159 name=publisher age=61060 idle=61060 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65797701 addr=10.18.44.236:44014 fd=164 name=subscriber age=61060 idle=61060 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760410 addr=10.18.44.236:53768 fd=53 name=cache age=66109 idle=66109 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688111 addr=10.18.44.236:54194 fd=661 name=locks age=81146 idle=81146 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688112 addr=10.18.44.236:54200 fd=662 name=publisher age=81146 idle=81146 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688113 addr=10.18.44.236:54206 fd=663 name=subscriber age=81146 idle=81146 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687864 addr=10.18.17.153:59160 fd=644 name=cache age=81153 idle=81153 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65818135 addr=10.18.33.201:56470 fd=128 name=publisher age=58882 idle=58882 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65818136 addr=10.18.33.201:56476 fd=166 name=subscriber age=58882 idle=58882 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687865 addr=10.18.17.153:59168 fd=645 name=subscriber age=81153 idle=81153 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687866 addr=10.18.17.153:59174 fd=646 name=locks age=81153 idle=81153 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687867 addr=10.18.17.153:59180 fd=647 name=publisher age=81153 idle=81153 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65886820 addr=10.18.33.201:58642 fd=205 name=cache age=52270 idle=52270 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441906 addr=10.18.13.92:57602 fd=271 name=cache age=3934 idle=3934 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441907 addr=10.18.13.92:57608 fd=272 name=subscriber age=3934 idle=3934 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441912 addr=10.18.13.92:57614 fd=270 name=locks age=3934 idle=3934 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441913 addr=10.18.13.92:57618 fd=277 name=publisher age=3934 idle=3934 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65886821 addr=10.18.33.201:58636 fd=206 name=locks age=52270 idle=52270 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65886822 addr=10.18.33.201:58648 fd=207 name=publisher age=52270 idle=52270 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66353306 addr=10.18.22.131:34332 fd=209 name=locks age=11085 idle=11085 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66353307 addr=10.18.22.131:34340 fd=214 name=cache age=11085 idle=11085 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66353308 addr=10.18.22.131:34344 fd=184 name=publisher age=11085 idle=11085 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66353309 addr=10.18.22.131:34350 fd=190 name=subscriber age=11085 idle=11085 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65886823 addr=10.18.33.201:58654 fd=208 name=subscriber age=52270 idle=52270 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772237 addr=10.18.22.131:43732 fd=114 name=locks age=64257 idle=64257 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772238 addr=10.18.22.131:43726 fd=115 name=cache age=64257 idle=64257 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772239 addr=10.18.22.131:43738 fd=116 name=subscriber age=64257 idle=64257 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788127 addr=10.18.4.128:47118 fd=149 name=locks age=62154 idle=62154 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65772240 addr=10.18.22.131:43746 fd=117 name=publisher age=64257 idle=64257 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66366519 addr=10.18.22.131:50386 fd=218 name=locks age=9945 idle=9945 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66366520 addr=10.18.22.131:50394 fd=219 name=cache age=9945 idle=9945 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66366521 addr=10.18.22.131:50400 fd=216 name=subscriber age=9945 idle=9945 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66366522 addr=10.18.22.131:50406 fd=220 name=publisher age=9945 idle=9945 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65698933 addr=10.18.4.128:48118 fd=56 name=cache age=79028 idle=79028 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65759603 addr=10.18.8.64:42794 fd=42 name=locks age=66251 idle=66251 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65781003 addr=10.18.22.131:52876 fd=136 name=cache age=63049 idle=63049 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65759604 addr=10.18.8.64:42790 fd=43 name=subscriber age=66251 idle=66251 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65759605 addr=10.18.8.64:42808 fd=44 name=cache age=66251 idle=66251 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65759606 addr=10.18.8.64:42802 fd=45 name=publisher age=66251 idle=66251 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65698934 addr=10.18.4.128:48122 fd=64 name=subscriber age=79028 idle=79028 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688106 addr=10.18.41.124:41790 fd=658 name=publisher age=81146 idle=81146 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688107 addr=10.18.41.124:41794 fd=659 name=locks age=81146 idle=81146 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688108 addr=10.18.41.124:41802 fd=660 name=subscriber age=81146 idle=81146 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536109 addr=10.18.96.90:52330 fd=447 name=subscriber age=305389 idle=305389 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65692871 addr=10.18.8.64:47314 fd=57 name=publisher age=80547 idle=80547 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65837095 addr=10.18.41.124:42552 fd=135 name=cache age=56962 idle=56962 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65837096 addr=10.18.41.124:42558 fd=185 name=publisher age=56962 idle=56961 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65837097 addr=10.18.41.124:42562 fd=189 name=locks age=56962 idle=56961 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65837098 addr=10.18.41.124:42568 fd=192 name=subscriber age=56962 idle=56961 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788124 addr=10.18.4.128:47098 fd=146 name=cache age=62154 idle=62154 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788125 addr=10.18.4.128:47104 fd=147 name=publisher age=62154 idle=62154 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788126 addr=10.18.4.128:47108 fd=148 name=subscriber age=62154 idle=62154 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65692872 addr=10.18.8.64:47320 fd=59 name=cache age=80547 idle=80547 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65692873 addr=10.18.8.64:47324 fd=60 name=locks age=80547 idle=80547 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65692874 addr=10.18.8.64:47332 fd=61 name=subscriber age=80547 idle=80547 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777194 addr=10.18.8.64:45570 fd=54 name=subscriber age=63542 idle=63542 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777195 addr=10.18.8.64:45578 fd=131 name=cache age=63542 idle=63542 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777196 addr=10.18.8.64:45590 fd=133 name=locks age=63542 idle=63542 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66428086 addr=10.18.44.236:47448 fd=242 name=cache age=4964 idle=4964 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66428087 addr=10.18.44.236:47452 fd=246 name=subscriber age=4964 idle=4964 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66428088 addr=10.18.44.236:47460 fd=247 name=publisher age=4964 idle=4964 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66428089 addr=10.18.44.236:47466 fd=248 name=locks age=4964 idle=4964 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65777197 addr=10.18.8.64:45584 fd=134 name=publisher age=63542 idle=63542 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65818130 addr=10.18.33.201:56458 fd=173 name=cache age=58883 idle=58882 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65818131 addr=10.18.33.201:56466 fd=174 name=locks age=58883 idle=58882 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702225 addr=10.18.41.124:53200 fd=67 name=locks age=78241 idle=78241 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702226 addr=10.18.41.124:53208 fd=68 name=publisher age=78241 idle=78241 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702227 addr=10.18.41.124:53214 fd=69 name=cache age=78241 idle=78241 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65702228 addr=10.18.41.124:53220 fd=70 name=subscriber age=78241 idle=78241 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536110 addr=10.18.96.90:52334 fd=448 name=publisher age=305389 idle=305389 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536111 addr=10.18.96.90:52338 fd=449 name=locks age=305389 idle=305389 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441743 addr=10.18.23.119:55878 fd=259 name=cache age=3946 idle=3946 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441744 addr=10.18.23.119:55886 fd=260 name=publisher age=3946 idle=3946 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441745 addr=10.18.23.119:55892 fd=261 name=subscriber age=3946 idle=3946 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441746 addr=10.18.23.119:55898 fd=262 name=locks age=3946 idle=3946 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=18951707 addr=10.18.83.50:39236 fd=151 name=locks age=4815327 idle=4815326 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688098 addr=10.18.41.124:41782 fd=656 name=cache age=81146 idle=81146 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=18951708 addr=10.18.83.50:39242 fd=25 name=publisher age=4815326 idle=4815326 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=18951705 addr=10.18.83.50:39228 fd=58 name=subscriber age=4815327 idle=4815327 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=18951706 addr=10.18.83.50:39224 fd=150 name=cache age=4815327 idle=4815327 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688254 addr=10.18.33.201:60056 fd=665 name=cache age=81141 idle=81141 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688257 addr=10.18.33.201:60064 fd=664 name=locks age=81141 idle=81141 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688258 addr=10.18.33.201:60070 fd=666 name=subscriber age=81141 idle=81141 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65688259 addr=10.18.33.201:60074 fd=667 name=publisher age=81141 idle=81141 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687836 addr=10.18.24.157:36980 fd=638 name=subscriber age=81154 idle=81154 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687837 addr=10.18.24.157:36974 fd=649 name=publisher age=81154 idle=81154 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65762843 addr=10.18.8.64:47768 fd=21 name=cache age=65714 idle=65714 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65762844 addr=10.18.8.64:47776 fd=109 name=locks age=65714 idle=65714 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65762845 addr=10.18.8.64:47788 fd=110 name=subscriber age=65714 idle=65714 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65762846 addr=10.18.8.64:47782 fd=111 name=publisher age=65714 idle=65714 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755042 addr=10.18.44.236:47584 fd=22 name=publisher age=67070 idle=67070 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755043 addr=10.18.44.236:47590 fd=31 name=subscriber age=67070 idle=67070 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755044 addr=10.18.44.236:47594 fd=36 name=locks age=67070 idle=67070 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66466440 addr=10.18.4.128:49866 fd=81 name=subscriber age=2094 idle=2094 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66466441 addr=10.18.4.128:49870 fd=80 name=locks age=2094 idle=2094 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66466442 addr=10.18.4.128:49876 fd=79 name=publisher age=2094 idle=2094 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66466443 addr=10.18.4.128:49882 fd=82 name=cache age=2094 idle=2094 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788555 addr=10.18.24.157:34954 fd=123 name=locks age=62102 idle=62102 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788556 addr=10.18.24.157:34960 fd=141 name=publisher age=62102 idle=62102 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788557 addr=10.18.24.157:34966 fd=152 name=subscriber age=62102 idle=62102 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65788558 addr=10.18.24.157:34972 fd=153 name=cache age=62102 idle=62102 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65755045 addr=10.18.44.236:47576 fd=37 name=cache age=67070 idle=67070 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687838 addr=10.18.24.157:36968 fd=650 name=locks age=81154 idle=81154 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65687839 addr=10.18.24.157:36964 fd=651 name=cache age=81154 idle=81154 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441924 addr=10.18.0.45:44620 fd=257 name=subscriber age=3934 idle=3934 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536129 addr=10.18.96.176:32996 fd=451 name=cache age=305389 idle=305389 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66368280 addr=10.18.17.153:37306 fd=227 name=locks age=9794 idle=9794 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66368281 addr=10.18.17.153:37314 fd=228 name=subscriber age=9794 idle=9794 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66368282 addr=10.18.17.153:37318 fd=229 name=publisher age=9794 idle=9794 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66368283 addr=10.18.17.153:37326 fd=230 name=cache age=9794 idle=9794 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65720871 addr=10.18.17.153:58340 fd=26 name=locks age=74238 idle=74238 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65720872 addr=10.18.17.153:58348 fd=27 name=cache age=74238 idle=74238 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66467131 addr=10.18.4.128:50482 fd=83 name=subscriber age=2043 idle=2043 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66467132 addr=10.18.4.128:50486 fd=23 name=cache age=2043 idle=2043 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66467133 addr=10.18.4.128:50494 fd=77 name=publisher age=2043 idle=2043 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66467134 addr=10.18.4.128:50498 fd=78 name=locks age=2043 idle=2043 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65720874 addr=10.18.17.153:58354 fd=29 name=publisher age=74238 idle=74238 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441893 addr=10.18.17.222:56136 fd=266 name=locks age=3936 idle=3936 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441894 addr=10.18.17.222:56144 fd=267 name=subscriber age=3936 idle=3936 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441895 addr=10.18.17.222:56150 fd=268 name=publisher age=3936 idle=3936 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441896 addr=10.18.17.222:56156 fd=269 name=cache age=3936 idle=3936 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65720875 addr=10.18.17.153:58360 fd=30 name=subscriber age=74238 idle=74238 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=60536130 addr=10.18.96.176:33002 fd=452 name=publisher age=305389 idle=305389 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66398943 addr=10.18.44.236:36838 fd=233 name=cache age=7276 idle=7276 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66398944 addr=10.18.44.236:36846 fd=234 name=publisher age=7276 idle=7276 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66398945 addr=10.18.44.236:36852 fd=235 name=locks age=7276 idle=7276 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66398946 addr=10.18.44.236:36856 fd=236 name=subscriber age=7276 idle=7276 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65760195 addr=10.18.8.64:43558 fd=39 name=locks age=66147 idle=66147 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65790643 addr=10.18.41.124:53862 fd=154 name=cache age=61866 idle=61866 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65790644 addr=10.18.41.124:53882 fd=160 name=locks age=61866 idle=61866 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65790645 addr=10.18.41.124:53876 fd=161 name=subscriber age=61866 idle=61866 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=65880315 addr=10.18.33.201:55062 fd=200 name=cache age=52864 idle=52864 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441920 addr=10.18.0.45:44604 fd=273 name=locks age=3934 idle=3934 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441921 addr=10.18.0.45:44614 fd=274 name=publisher age=3934 idle=3934 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441922 addr=10.18.0.45:44608 fd=275 name=cache age=3934 idle=3934 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66426004 addr=10.18.44.236:44308 fd=221 name=publisher age=5121 idle=5121 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66426006 addr=10.18.44.236:44314 fd=243 name=locks age=5121 idle=5121 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66426007 addr=10.18.44.236:44320 fd=244 name=subscriber age=5121 idle=5121 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441954 addr=10.18.39.102:56012 fd=278 name=publisher age=3933 idle=3933 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441955 addr=10.18.39.102:56020 fd=279 name=locks age=3933 idle=3933 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default
id=66441956 addr=10.18.39.102:56026 fd=280 name=subscriber age=3933 idle=3933 flags=r db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=readonly user=default

@sikmir
Copy link

sikmir commented Oct 27, 2021

Maybe somehow related to duplication of connection introduced in 4.9.5 (#867):
https://github.com/luin/ioredis/blob/77ad094183c99deb41f5f4795d0e0b3a39931f50/lib/cluster/index.ts#L799-L810

@Cellule
Copy link
Author

Cellule commented Oct 27, 2021

Maybe somehow related to duplication of connection introduced in 4.9.5 (#867):

Maybe I'm reading this wrong, but should it seems the duplicate connection should have a different connection name, which I'm not seeing in the client list results

@sikmir
Copy link

sikmir commented Oct 27, 2021

Maybe I'm reading this wrong, but should it seems the duplicate connection should have a different connection name, which I'm not seeing in the client list results

Yes, you are right, connection name should contain "refresher", but does not in your case.

@Cellule
Copy link
Author

Cellule commented Oct 27, 2021

I've just instrumented our code to log on +node -node node error events.
Might take a bit of time to be deployed in production and have something to look at, but I'm hoping it'll help pinpoint what is happening
I'm considering using DEBUG=ioredis:cluster:* but was want to see if a less verbose logging can help first

@Cellule
Copy link
Author

Cellule commented Oct 29, 2021

We are logging tons of node error: timeout events and quite a few +node events.
No -node so far
There are wayyy more timeout errors than +node events and I can't quite say for sure if the 2 are related

@Cellule
Copy link
Author

Cellule commented Oct 29, 2021

I added DEBUG:ioredis:* on 1 of our servers
I see a lot of the following which I believe are just the bookeeping to know which cluster node are master/slave and which slot they are configured to.
Doing this every 5s feels a bit aggressive IMO, it is configurable so I might change it, unless there's a good reason to do it so often (I guess in case something changes unexpectedly on the cluste)

ioredis:cluster refreshing slot caches... (triggered by "slotsRefreshInterval" option)
ioredis:cluster getting slot cache from mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379
ioredis:redis status[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):locks)]: [empty] -> wait
ioredis:redis status[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):locks)]: wait -> connecting
ioredis:redis queue command[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):locks)]: 0 -> cluster([ 'slots' ])
ioredis:redis status[10.18.81.137:6379 (ioredis-cluster(refresher):locks)]: connecting -> connect
ioredis:redis status[10.18.81.137:6379 (ioredis-cluster(refresher):locks)]: connect -> ready
ioredis:connection set the connection name [ioredis-cluster(refresher):locks]
ioredis:redis write command[10.18.81.137:6379 (ioredis-cluster(refresher):locks)]: 0 -> client([ 'setname', 'ioredis-cluster(refresher):locks' ])
ioredis:connection send 1 commands in offline queue
ioredis:redis write command[10.18.81.137:6379 (ioredis-cluster(refresher):locks)]: 0 -> cluster([ 'slots' ])
ioredis:cluster cluster slots result count: 2
ioredis:cluster cluster slots result [0]: slots 8192~16383 served by [ 'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379', 'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379' ]
ioredis:cluster cluster slots result [1]: slots 0~8191 served by [ 'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379', 'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379' ]
ioredis:cluster:connectionPool Reset with [ { host: 'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com', port: 6379, readOnly: false }, { host: 'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com', port: 6379, readOnly: true }, { host: 'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com', port: 6379, readOnly: false }, { host: 'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com', port: 6379, readOnly: true } ]
ioredis:redis status[10.18.81.137:6379 (ioredis-cluster(refresher):locks)]: ready -> close
ioredis:connection skip reconnecting since the connection is manually closed.
ioredis:redis status[10.18.81.137:6379 (ioredis-cluster(refresher):locks)]: close -> end

@Cellule
Copy link
Author

Cellule commented Oct 29, 2021

Here's logs of the booting sequence

Not sure if a problem but I find it interesting that it initially connects to all nodes as master and waits for first refresh to determine which really are master and which are slaves.
I'm guessing there can't be any traffic during that very very short time span so it's probably fine

2021-10-29 17:41:14.520 +00:00: 2021-10-29T17:41:14.520Z ioredis:cluster status: [empty] -> connecting
--
2021-10-29 17:41:14.521 +00:00: 2021-10-29T17:41:14.521Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.521 +00:00: 2021-10-29T17:41:14.521Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.521 +00:00: 2021-10-29T17:41:14.521Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.521 +00:00: 2021-10-29T17:41:14.521Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.522 +00:00: 2021-10-29T17:41:14.522Z ioredis:cluster status: [empty] -> connecting
2021-10-29 17:41:14.522 +00:00: 2021-10-29T17:41:14.522Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.522 +00:00: 2021-10-29T17:41:14.522Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.522 +00:00: 2021-10-29T17:41:14.522Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.522 +00:00: 2021-10-29T17:41:14.522Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.523 +00:00: 2021-10-29T17:41:14.523Z ioredis:cluster status: [empty] -> connecting
2021-10-29 17:41:14.523 +00:00: 2021-10-29T17:41:14.523Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.523 +00:00: 2021-10-29T17:41:14.523Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.524 +00:00: 2021-10-29T17:41:14.524Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.524 +00:00: 2021-10-29T17:41:14.524Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.524 +00:00: 2021-10-29T17:41:14.524Z ioredis:cluster status: [empty] -> connecting
2021-10-29 17:41:14.524 +00:00: 2021-10-29T17:41:14.524Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.524 +00:00: 2021-10-29T17:41:14.524Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.524 +00:00: 2021-10-29T17:41:14.524Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:14.524 +00:00: 2021-10-29T17:41:14.524Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com
2021-10-29 17:41:15.464 +00:00: 2021-10-29T17:41:15.464Z ioredis:cluster:connectionPool Reset with [  {    port: 6379,    host: 'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  },  {    port: 6379,    host: 'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  },  {    port: 6379,    host: 'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  },  {    port: 6379,    host: 'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  }]
2021-10-29 17:41:15.464 +00:00: 2021-10-29T17:41:15.464Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.465 +00:00: 2021-10-29T17:41:15.465Z ioredis:redis status[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (cache)]: [empty] -> wait
2021-10-29 17:41:15.472 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.473 +00:00: 2021-10-29T17:41:15.473Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.473 +00:00: 2021-10-29T17:41:15.473Z ioredis:redis status[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (cache)]: [empty] -> wait
2021-10-29 17:41:15.475 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.475 +00:00: 2021-10-29T17:41:15.475Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.476 +00:00: 2021-10-29T17:41:15.475Z ioredis:redis status[mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (cache)]: [empty] -> wait
2021-10-29 17:41:15.477 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.478 +00:00: 2021-10-29T17:41:15.478Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.478 +00:00: 2021-10-29T17:41:15.478Z ioredis:redis status[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (cache)]: [empty] -> wait
2021-10-29 17:41:15.480 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.480 +00:00: 2021-10-29T17:41:15.480Z ioredis:cluster getting slot cache from mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379
2021-10-29 17:41:15.481 +00:00: 2021-10-29T17:41:15.481Z ioredis:redis status[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):cache)]: [empty] -> wait
2021-10-29 17:41:15.483 +00:00: 2021-10-29T17:41:15.482Z ioredis:redis status[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):cache)]: wait -> connecting
2021-10-29 17:41:15.483 +00:00: 2021-10-29T17:41:15.483Z ioredis:redis queue command[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):cache)]: 0 -> cluster([ 'slots' ])
2021-10-29 17:41:15.484 +00:00: 2021-10-29T17:41:15.484Z ioredis:cluster:subscriber selected a subscriber mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379
2021-10-29 17:41:15.484 +00:00: 2021-10-29T17:41:15.484Z ioredis:redis status[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(subscriber):cache)]: [empty] -> wait
2021-10-29 17:41:15.484 +00:00: 2021-10-29T17:41:15.484Z ioredis:cluster:subscriber started
2021-10-29 17:41:15.484 +00:00: 2021-10-29T17:41:15.484Z ioredis:cluster:connectionPool Reset with [  {    port: 6379,    host: 'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  },  {    port: 6379,    host: 'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  },  {    port: 6379,    host: 'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  },  {    port: 6379,    host: 'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  }]
2021-10-29 17:41:15.484 +00:00: 2021-10-29T17:41:15.484Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.485 +00:00: 2021-10-29T17:41:15.485Z ioredis:redis status[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (locks)]: [empty] -> wait
2021-10-29 17:41:15.503 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.503 +00:00: 2021-10-29T17:41:15.503Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.504 +00:00: 2021-10-29T17:41:15.504Z ioredis:redis status[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (locks)]: [empty] -> wait
2021-10-29 17:41:15.506 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.506 +00:00: 2021-10-29T17:41:15.506Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.507 +00:00: 2021-10-29T17:41:15.507Z ioredis:redis status[mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (locks)]: [empty] -> wait
2021-10-29 17:41:15.509 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.509 +00:00: 2021-10-29T17:41:15.509Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.509 +00:00: 2021-10-29T17:41:15.509Z ioredis:redis status[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (locks)]: [empty] -> wait
2021-10-29 17:41:15.511 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.512 +00:00: 2021-10-29T17:41:15.512Z ioredis:cluster getting slot cache from mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379
2021-10-29 17:41:15.512 +00:00: 2021-10-29T17:41:15.512Z ioredis:redis status[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):locks)]: [empty] -> wait
2021-10-29 17:41:15.513 +00:00: 2021-10-29T17:41:15.513Z ioredis:redis status[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):locks)]: wait -> connecting
2021-10-29 17:41:15.513 +00:00: 2021-10-29T17:41:15.513Z ioredis:redis queue command[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):locks)]: 0 -> cluster([ 'slots' ])
2021-10-29 17:41:15.513 +00:00: 2021-10-29T17:41:15.513Z ioredis:cluster:subscriber selected a subscriber mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379
2021-10-29 17:41:15.513 +00:00: 2021-10-29T17:41:15.513Z ioredis:redis status[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(subscriber):locks)]: [empty] -> wait
2021-10-29 17:41:15.513 +00:00: 2021-10-29T17:41:15.513Z ioredis:cluster:subscriber started
2021-10-29 17:41:15.514 +00:00: 2021-10-29T17:41:15.514Z ioredis:cluster:connectionPool Reset with [  {    port: 6379,    host: 'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  },  {    port: 6379,    host: 'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  },  {    port: 6379,    host: 'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  },  {    port: 6379,    host: 'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  }]
2021-10-29 17:41:15.514 +00:00: 2021-10-29T17:41:15.514Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.514 +00:00: 2021-10-29T17:41:15.514Z ioredis:redis status[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (publisher)]: [empty] -> wait
2021-10-29 17:41:15.516 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.516 +00:00: 2021-10-29T17:41:15.516Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.517 +00:00: 2021-10-29T17:41:15.517Z ioredis:redis status[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (publisher)]: [empty] -> wait
2021-10-29 17:41:15.519 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.519 +00:00: 2021-10-29T17:41:15.519Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.519 +00:00: 2021-10-29T17:41:15.519Z ioredis:redis status[mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (publisher)]: [empty] -> wait
2021-10-29 17:41:15.521 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.522 +00:00: 2021-10-29T17:41:15.522Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.522 +00:00: 2021-10-29T17:41:15.522Z ioredis:redis status[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (publisher)]: [empty] -> wait
2021-10-29 17:41:15.524 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.524 +00:00: 2021-10-29T17:41:15.524Z ioredis:cluster getting slot cache from mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379
2021-10-29 17:41:15.525 +00:00: 2021-10-29T17:41:15.525Z ioredis:redis status[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):publisher)]: [empty] -> wait
2021-10-29 17:41:15.525 +00:00: 2021-10-29T17:41:15.525Z ioredis:redis status[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):publisher)]: wait -> connecting
2021-10-29 17:41:15.525 +00:00: 2021-10-29T17:41:15.525Z ioredis:redis queue command[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):publisher)]: 0 -> cluster([ 'slots' ])
2021-10-29 17:41:15.525 +00:00: 2021-10-29T17:41:15.525Z ioredis:cluster:subscriber selected a subscriber mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379
2021-10-29 17:41:15.525 +00:00: 2021-10-29T17:41:15.525Z ioredis:redis status[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(subscriber):publisher)]: [empty] -> wait
2021-10-29 17:41:15.525 +00:00: 2021-10-29T17:41:15.525Z ioredis:cluster:subscriber started
2021-10-29 17:41:15.526 +00:00: 2021-10-29T17:41:15.526Z ioredis:cluster:connectionPool Reset with [  {    port: 6379,    host: 'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  },  {    port: 6379,    host: 'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  },  {    port: 6379,    host: 'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  },  {    port: 6379,    host: 'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com'  }]
2021-10-29 17:41:15.526 +00:00: 2021-10-29T17:41:15.526Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.526 +00:00: 2021-10-29T17:41:15.526Z ioredis:redis status[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (subscriber)]: [empty] -> wait
2021-10-29 17:41:15.529 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.529 +00:00: 2021-10-29T17:41:15.529Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.529 +00:00: 2021-10-29T17:41:15.529Z ioredis:redis status[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (subscriber)]: [empty] -> wait
2021-10-29 17:41:15.531 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.531 +00:00: 2021-10-29T17:41:15.531Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.532 +00:00: 2021-10-29T17:41:15.532Z ioredis:redis status[mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (subscriber)]: [empty] -> wait
2021-10-29 17:41:15.534 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.534 +00:00: 2021-10-29T17:41:15.534Z ioredis:cluster:connectionPool Connecting to mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 as master
2021-10-29 17:41:15.534 +00:00: 2021-10-29T17:41:15.534Z ioredis:redis status[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (subscriber)]: [empty] -> wait
2021-10-29 17:41:15.536 +00:00: info:     ioredis:cluster:node
2021-10-29 17:41:15.536 +00:00: 2021-10-29T17:41:15.536Z ioredis:cluster getting slot cache from mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379
2021-10-29 17:41:15.537 +00:00: 2021-10-29T17:41:15.537Z ioredis:redis status[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):subscriber)]: [empty] -> wait
2021-10-29 17:41:15.537 +00:00: 2021-10-29T17:41:15.537Z ioredis:redis status[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):subscriber)]: wait -> connecting
2021-10-29 17:41:15.537 +00:00: 2021-10-29T17:41:15.537Z ioredis:redis queue command[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(refresher):subscriber)]: 0 -> cluster([ 'slots' ])
2021-10-29 17:41:15.537 +00:00: 2021-10-29T17:41:15.537Z ioredis:cluster:subscriber selected a subscriber mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379
2021-10-29 17:41:15.538 +00:00: 2021-10-29T17:41:15.537Z ioredis:redis status[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (ioredis-cluster(subscriber):subscriber)]: [empty] -> wait
2021-10-29 17:41:15.538 +00:00: 2021-10-29T17:41:15.538Z ioredis:cluster:subscriber started
2021-10-29 17:41:15.631 +00:00: 2021-10-29T17:41:15.631Z ioredis:redis status[10.18.75.185:6379 (ioredis-cluster(refresher):subscriber)]: connecting -> connect
2021-10-29 17:41:15.632 +00:00: 2021-10-29T17:41:15.632Z ioredis:redis status[10.18.75.185:6379 (ioredis-cluster(refresher):subscriber)]: connect -> ready
2021-10-29 17:41:15.632 +00:00: 2021-10-29T17:41:15.632Z ioredis:connection set the connection name [ioredis-cluster(refresher):subscriber]
2021-10-29 17:41:15.632 +00:00: 2021-10-29T17:41:15.632Z ioredis:redis write command[10.18.75.185:6379 (ioredis-cluster(refresher):subscriber)]: 0 -> client([ 'setname', 'ioredis-cluster(refresher):subscriber' ])
2021-10-29 17:41:15.633 +00:00: 2021-10-29T17:41:15.633Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.633 +00:00: 2021-10-29T17:41:15.633Z ioredis:redis write command[10.18.75.185:6379 (ioredis-cluster(refresher):subscriber)]: 0 -> cluster([ 'slots' ])
2021-10-29 17:41:15.635 +00:00: 2021-10-29T17:41:15.635Z ioredis:redis status[10.18.85.21:6379 (ioredis-cluster(refresher):locks)]: connecting -> connect
2021-10-29 17:41:15.635 +00:00: 2021-10-29T17:41:15.635Z ioredis:redis status[10.18.85.21:6379 (ioredis-cluster(refresher):locks)]: connect -> ready
2021-10-29 17:41:15.635 +00:00: 2021-10-29T17:41:15.635Z ioredis:connection set the connection name [ioredis-cluster(refresher):locks]
2021-10-29 17:41:15.635 +00:00: 2021-10-29T17:41:15.635Z ioredis:redis write command[10.18.85.21:6379 (ioredis-cluster(refresher):locks)]: 0 -> client([ 'setname', 'ioredis-cluster(refresher):locks' ])
2021-10-29 17:41:15.635 +00:00: 2021-10-29T17:41:15.635Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.636 +00:00: 2021-10-29T17:41:15.635Z ioredis:redis write command[10.18.85.21:6379 (ioredis-cluster(refresher):locks)]: 0 -> cluster([ 'slots' ])
2021-10-29 17:41:15.637 +00:00: 2021-10-29T17:41:15.637Z ioredis:redis status[10.18.81.137:6379 (ioredis-cluster(refresher):publisher)]: connecting -> connect
2021-10-29 17:41:15.637 +00:00: 2021-10-29T17:41:15.637Z ioredis:redis status[10.18.81.137:6379 (ioredis-cluster(refresher):publisher)]: connect -> ready
2021-10-29 17:41:15.637 +00:00: 2021-10-29T17:41:15.637Z ioredis:connection set the connection name [ioredis-cluster(refresher):publisher]
2021-10-29 17:41:15.637 +00:00: 2021-10-29T17:41:15.637Z ioredis:redis write command[10.18.81.137:6379 (ioredis-cluster(refresher):publisher)]: 0 -> client([ 'setname', 'ioredis-cluster(refresher):publisher' ])
2021-10-29 17:41:15.637 +00:00: 2021-10-29T17:41:15.637Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.638 +00:00: 2021-10-29T17:41:15.638Z ioredis:redis write command[10.18.81.137:6379 (ioredis-cluster(refresher):publisher)]: 0 -> cluster([ 'slots' ])
2021-10-29 17:41:15.639 +00:00: 2021-10-29T17:41:15.639Z ioredis:redis status[10.18.85.21:6379 (ioredis-cluster(refresher):cache)]: connecting -> connect
2021-10-29 17:41:15.639 +00:00: 2021-10-29T17:41:15.639Z ioredis:redis status[10.18.85.21:6379 (ioredis-cluster(refresher):cache)]: connect -> ready
2021-10-29 17:41:15.639 +00:00: 2021-10-29T17:41:15.639Z ioredis:connection set the connection name [ioredis-cluster(refresher):cache]
2021-10-29 17:41:15.639 +00:00: 2021-10-29T17:41:15.639Z ioredis:redis write command[10.18.85.21:6379 (ioredis-cluster(refresher):cache)]: 0 -> client([ 'setname', 'ioredis-cluster(refresher):cache' ])
2021-10-29 17:41:15.640 +00:00: 2021-10-29T17:41:15.640Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.640 +00:00: 2021-10-29T17:41:15.640Z ioredis:redis write command[10.18.85.21:6379 (ioredis-cluster(refresher):cache)]: 0 -> cluster([ 'slots' ])
2021-10-29 17:41:15.642 +00:00: 2021-10-29T17:41:15.642Z ioredis:cluster cluster slots result count: 2
2021-10-29 17:41:15.643 +00:00: 2021-10-29T17:41:15.642Z ioredis:cluster cluster slots result [0]: slots 8192~16383 served by [  'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379',  'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379']
2021-10-29 17:41:15.644 +00:00: 2021-10-29T17:41:15.644Z ioredis:cluster cluster slots result [1]: slots 0~8191 served by [  'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379',  'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379']
2021-10-29 17:41:15.657 +00:00: 2021-10-29T17:41:15.657Z ioredis:cluster:connectionPool Reset with [  {    host: 'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: false  },  {    host: 'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: true  },  {    host: 'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: false  },  {    host: 'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: true  }]
2021-10-29 17:41:15.658 +00:00: 2021-10-29T17:41:15.658Z ioredis:cluster:connectionPool Change role of mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 to slave
2021-10-29 17:41:15.658 +00:00: 2021-10-29T17:41:15.658Z ioredis:redis status[mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (locks)]: wait -> connecting
2021-10-29 17:41:15.658 +00:00: 2021-10-29T17:41:15.658Z ioredis:redis queue command[mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (locks)]: 0 -> readonly([])
2021-10-29 17:41:15.658 +00:00: 2021-10-29T17:41:15.658Z ioredis:cluster:connectionPool Change role of mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 to slave
2021-10-29 17:41:15.658 +00:00: 2021-10-29T17:41:15.658Z ioredis:redis status[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (locks)]: wait -> connecting
2021-10-29 17:41:15.658 +00:00: 2021-10-29T17:41:15.658Z ioredis:redis queue command[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (locks)]: 0 -> readonly([])
2021-10-29 17:41:15.659 +00:00: 2021-10-29T17:41:15.659Z ioredis:cluster status: connecting -> connect
2021-10-29 17:41:15.659 +00:00: 2021-10-29T17:41:15.659Z ioredis:redis status[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (locks)]: wait -> connecting
2021-10-29 17:41:15.660 +00:00: 2021-10-29T17:41:15.660Z ioredis:redis queue command[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (locks)]: 0 -> cluster([ 'info' ])
2021-10-29 17:41:15.662 +00:00: 2021-10-29T17:41:15.662Z ioredis:cluster cluster slots result count: 2
2021-10-29 17:41:15.663 +00:00: 2021-10-29T17:41:15.663Z ioredis:cluster cluster slots result [0]: slots 8192~16383 served by [  'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379',  'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379']
2021-10-29 17:41:15.664 +00:00: 2021-10-29T17:41:15.664Z ioredis:cluster cluster slots result [1]: slots 0~8191 served by [  'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379',  'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379']
2021-10-29 17:41:15.679 +00:00: 2021-10-29T17:41:15.679Z ioredis:cluster:connectionPool Reset with [  {    host: 'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: false  },  {    host: 'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: true  },  {    host: 'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: false  },  {    host: 'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: true  }]
2021-10-29 17:41:15.679 +00:00: 2021-10-29T17:41:15.679Z ioredis:cluster:connectionPool Change role of mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 to slave
2021-10-29 17:41:15.679 +00:00: 2021-10-29T17:41:15.679Z ioredis:redis status[mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (publisher)]: wait -> connecting
2021-10-29 17:41:15.680 +00:00: 2021-10-29T17:41:15.680Z ioredis:redis queue command[mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (publisher)]: 0 -> readonly([])
2021-10-29 17:41:15.680 +00:00: 2021-10-29T17:41:15.680Z ioredis:cluster:connectionPool Change role of mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 to slave
2021-10-29 17:41:15.680 +00:00: 2021-10-29T17:41:15.680Z ioredis:redis status[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (publisher)]: wait -> connecting
2021-10-29 17:41:15.680 +00:00: 2021-10-29T17:41:15.680Z ioredis:redis queue command[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (publisher)]: 0 -> readonly([])
2021-10-29 17:41:15.680 +00:00: 2021-10-29T17:41:15.680Z ioredis:cluster status: connecting -> connect
2021-10-29 17:41:15.680 +00:00: 2021-10-29T17:41:15.680Z ioredis:redis status[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (publisher)]: wait -> connecting
2021-10-29 17:41:15.680 +00:00: 2021-10-29T17:41:15.680Z ioredis:redis queue command[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (publisher)]: 0 -> cluster([ 'info' ])
2021-10-29 17:41:15.682 +00:00: 2021-10-29T17:41:15.682Z ioredis:cluster cluster slots result count: 2
2021-10-29 17:41:15.682 +00:00: 2021-10-29T17:41:15.682Z ioredis:cluster cluster slots result [0]: slots 8192~16383 served by [  'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379',  'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379']
2021-10-29 17:41:15.704 +00:00: 2021-10-29T17:41:15.704Z ioredis:cluster cluster slots result [1]: slots 0~8191 served by [  'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379',  'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379']
2021-10-29 17:41:15.719 +00:00: 2021-10-29T17:41:15.719Z ioredis:cluster:connectionPool Reset with [  {    host: 'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: false  },  {    host: 'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: true  },  {    host: 'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: false  },  {    host: 'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: true  }]
2021-10-29 17:41:15.720 +00:00: 2021-10-29T17:41:15.720Z ioredis:cluster:connectionPool Change role of mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 to slave
2021-10-29 17:41:15.720 +00:00: 2021-10-29T17:41:15.720Z ioredis:redis status[mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (cache)]: wait -> connecting
2021-10-29 17:41:15.720 +00:00: 2021-10-29T17:41:15.720Z ioredis:redis queue command[mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (cache)]: 0 -> readonly([])
2021-10-29 17:41:15.720 +00:00: 2021-10-29T17:41:15.720Z ioredis:cluster:connectionPool Change role of mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 to slave
2021-10-29 17:41:15.720 +00:00: 2021-10-29T17:41:15.720Z ioredis:redis status[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (cache)]: wait -> connecting
2021-10-29 17:41:15.720 +00:00: 2021-10-29T17:41:15.720Z ioredis:redis queue command[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (cache)]: 0 -> readonly([])
2021-10-29 17:41:15.720 +00:00: 2021-10-29T17:41:15.720Z ioredis:cluster status: connecting -> connect
2021-10-29 17:41:15.721 +00:00: 2021-10-29T17:41:15.721Z ioredis:redis status[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (cache)]: wait -> connecting
2021-10-29 17:41:15.721 +00:00: 2021-10-29T17:41:15.721Z ioredis:redis queue command[mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (cache)]: 0 -> cluster([ 'info' ])
2021-10-29 17:41:15.748 +00:00: 2021-10-29T17:41:15.748Z ioredis:cluster cluster slots result count: 2
2021-10-29 17:41:15.748 +00:00: 2021-10-29T17:41:15.748Z ioredis:cluster cluster slots result [0]: slots 0~8191 served by [  'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379',  'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379']
2021-10-29 17:41:15.748 +00:00: 2021-10-29T17:41:15.748Z ioredis:cluster cluster slots result [1]: slots 8192~16383 served by [  'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379',  'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379']
2021-10-29 17:41:15.761 +00:00: 2021-10-29T17:41:15.761Z ioredis:cluster:connectionPool Reset with [  {    host: 'mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: false  },  {    host: 'mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: true  },  {    host: 'mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: false  },  {    host: 'mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com',    port: 6379,    readOnly: true  }]
2021-10-29 17:41:15.762 +00:00: 2021-10-29T17:41:15.762Z ioredis:cluster:connectionPool Change role of mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 to slave
2021-10-29 17:41:15.762 +00:00: 2021-10-29T17:41:15.762Z ioredis:redis status[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (subscriber)]: wait -> connecting
2021-10-29 17:41:15.762 +00:00: 2021-10-29T17:41:15.762Z ioredis:redis queue command[mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (subscriber)]: 0 -> readonly([])
2021-10-29 17:41:15.762 +00:00: 2021-10-29T17:41:15.762Z ioredis:cluster:connectionPool Change role of mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 to slave
2021-10-29 17:41:15.762 +00:00: 2021-10-29T17:41:15.762Z ioredis:redis status[mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (subscriber)]: wait -> connecting
2021-10-29 17:41:15.762 +00:00: 2021-10-29T17:41:15.762Z ioredis:redis queue command[mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (subscriber)]: 0 -> readonly([])
2021-10-29 17:41:15.762 +00:00: 2021-10-29T17:41:15.762Z ioredis:cluster status: connecting -> connect
2021-10-29 17:41:15.762 +00:00: 2021-10-29T17:41:15.762Z ioredis:redis status[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (subscriber)]: wait -> connecting
2021-10-29 17:41:15.763 +00:00: 2021-10-29T17:41:15.763Z ioredis:redis queue command[mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com:6379 (subscriber)]: 0 -> cluster([ 'info' ])
2021-10-29 17:41:15.768 +00:00: 2021-10-29T17:41:15.768Z ioredis:redis status[10.18.85.21:6379 (ioredis-cluster(refresher):cache)]: ready -> close
2021-10-29 17:41:15.768 +00:00: 2021-10-29T17:41:15.768Z ioredis:connection skip reconnecting since the connection is manually closed.
2021-10-29 17:41:15.768 +00:00: 2021-10-29T17:41:15.768Z ioredis:redis status[10.18.85.21:6379 (ioredis-cluster(refresher):cache)]: close -> end
2021-10-29 17:41:15.769 +00:00: 2021-10-29T17:41:15.768Z ioredis:redis status[10.18.81.137:6379 (ioredis-cluster(refresher):publisher)]: ready -> close
2021-10-29 17:41:15.769 +00:00: 2021-10-29T17:41:15.769Z ioredis:connection skip reconnecting since the connection is manually closed.
2021-10-29 17:41:15.769 +00:00: 2021-10-29T17:41:15.769Z ioredis:redis status[10.18.81.137:6379 (ioredis-cluster(refresher):publisher)]: close -> end
2021-10-29 17:41:15.769 +00:00: 2021-10-29T17:41:15.769Z ioredis:redis status[10.18.85.21:6379 (ioredis-cluster(refresher):locks)]: ready -> close
2021-10-29 17:41:15.769 +00:00: 2021-10-29T17:41:15.769Z ioredis:connection skip reconnecting since the connection is manually closed.
2021-10-29 17:41:15.769 +00:00: 2021-10-29T17:41:15.769Z ioredis:redis status[10.18.85.21:6379 (ioredis-cluster(refresher):locks)]: close -> end
2021-10-29 17:41:15.803 +00:00: 2021-10-29T17:41:15.803Z ioredis:redis status[10.18.75.185:6379 (ioredis-cluster(refresher):subscriber)]: ready -> close
2021-10-29 17:41:15.803 +00:00: 2021-10-29T17:41:15.803Z ioredis:connection skip reconnecting since the connection is manually closed.
2021-10-29 17:41:15.803 +00:00: 2021-10-29T17:41:15.803Z ioredis:redis status[10.18.75.185:6379 (ioredis-cluster(refresher):subscriber)]: close -> end
2021-10-29 17:41:15.805 +00:00: 2021-10-29T17:41:15.805Z ioredis:redis status[10.18.75.185:6379 (locks)]: connecting -> connect
2021-10-29 17:41:15.805 +00:00: 2021-10-29T17:41:15.805Z ioredis:redis write command[10.18.75.185:6379 (locks)]: 0 -> info([])
2021-10-29 17:41:15.807 +00:00: 2021-10-29T17:41:15.807Z ioredis:redis status[10.18.81.137:6379 (locks)]: connecting -> connect
2021-10-29 17:41:15.807 +00:00: 2021-10-29T17:41:15.807Z ioredis:redis write command[10.18.81.137:6379 (locks)]: 0 -> info([])
2021-10-29 17:41:15.808 +00:00: 2021-10-29T17:41:15.808Z ioredis:redis status[10.18.62.156:6379 (locks)]: connecting -> connect
2021-10-29 17:41:15.809 +00:00: 2021-10-29T17:41:15.809Z ioredis:redis write command[10.18.62.156:6379 (locks)]: 0 -> info([])
2021-10-29 17:41:15.810 +00:00: 2021-10-29T17:41:15.810Z ioredis:redis status[10.18.62.156:6379 (publisher)]: connecting -> connect
2021-10-29 17:41:15.810 +00:00: 2021-10-29T17:41:15.810Z ioredis:redis write command[10.18.62.156:6379 (publisher)]: 0 -> info([])
2021-10-29 17:41:15.811 +00:00: 2021-10-29T17:41:15.811Z ioredis:redis status[10.18.75.185:6379 (publisher)]: connecting -> connect
2021-10-29 17:41:15.812 +00:00: 2021-10-29T17:41:15.811Z ioredis:redis write command[10.18.75.185:6379 (publisher)]: 0 -> info([])
2021-10-29 17:41:15.813 +00:00: 2021-10-29T17:41:15.813Z ioredis:redis status[10.18.85.21:6379 (publisher)]: connecting -> connect
2021-10-29 17:41:15.813 +00:00: 2021-10-29T17:41:15.813Z ioredis:redis write command[10.18.85.21:6379 (publisher)]: 0 -> info([])
2021-10-29 17:41:15.818 +00:00: 2021-10-29T17:41:15.817Z ioredis:redis status[10.18.75.185:6379 (locks)]: connect -> ready
2021-10-29 17:41:15.818 +00:00: 2021-10-29T17:41:15.818Z ioredis:connection set the connection name [locks]
2021-10-29 17:41:15.818 +00:00: 2021-10-29T17:41:15.818Z ioredis:redis write command[10.18.75.185:6379 (locks)]: 0 -> client([ 'setname', 'locks' ])
2021-10-29 17:41:15.818 +00:00: 2021-10-29T17:41:15.818Z ioredis:connection set the connection to readonly mode
2021-10-29 17:41:15.818 +00:00: 2021-10-29T17:41:15.818Z ioredis:redis write command[10.18.75.185:6379 (locks)]: 0 -> readonly([])
2021-10-29 17:41:15.818 +00:00: 2021-10-29T17:41:15.818Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.818 +00:00: 2021-10-29T17:41:15.818Z ioredis:redis write command[10.18.75.185:6379 (locks)]: 0 -> readonly([])
2021-10-29 17:41:15.819 +00:00: 2021-10-29T17:41:15.819Z ioredis:redis status[10.18.81.137:6379 (locks)]: connect -> ready
2021-10-29 17:41:15.819 +00:00: 2021-10-29T17:41:15.819Z ioredis:connection set the connection name [locks]
2021-10-29 17:41:15.819 +00:00: 2021-10-29T17:41:15.819Z ioredis:redis write command[10.18.81.137:6379 (locks)]: 0 -> client([ 'setname', 'locks' ])
2021-10-29 17:41:15.819 +00:00: 2021-10-29T17:41:15.819Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.820 +00:00: 2021-10-29T17:41:15.820Z ioredis:redis write command[10.18.81.137:6379 (locks)]: 0 -> cluster([ 'info' ])
2021-10-29 17:41:15.820 +00:00: 2021-10-29T17:41:15.820Z ioredis:redis status[10.18.62.156:6379 (locks)]: connect -> ready
2021-10-29 17:41:15.820 +00:00: 2021-10-29T17:41:15.820Z ioredis:connection set the connection name [locks]
2021-10-29 17:41:15.821 +00:00: 2021-10-29T17:41:15.820Z ioredis:redis write command[10.18.62.156:6379 (locks)]: 0 -> client([ 'setname', 'locks' ])
2021-10-29 17:41:15.821 +00:00: 2021-10-29T17:41:15.821Z ioredis:connection set the connection to readonly mode
2021-10-29 17:41:15.821 +00:00: 2021-10-29T17:41:15.821Z ioredis:redis write command[10.18.62.156:6379 (locks)]: 0 -> readonly([])
2021-10-29 17:41:15.821 +00:00: 2021-10-29T17:41:15.821Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.821 +00:00: 2021-10-29T17:41:15.821Z ioredis:redis write command[10.18.62.156:6379 (locks)]: 0 -> readonly([])
2021-10-29 17:41:15.822 +00:00: 2021-10-29T17:41:15.822Z ioredis:redis status[10.18.62.156:6379 (publisher)]: connect -> ready
2021-10-29 17:41:15.822 +00:00: 2021-10-29T17:41:15.822Z ioredis:connection set the connection name [publisher]
2021-10-29 17:41:15.822 +00:00: 2021-10-29T17:41:15.822Z ioredis:redis write command[10.18.62.156:6379 (publisher)]: 0 -> client([ 'setname', 'publisher' ])
2021-10-29 17:41:15.822 +00:00: 2021-10-29T17:41:15.822Z ioredis:connection set the connection to readonly mode
2021-10-29 17:41:15.822 +00:00: 2021-10-29T17:41:15.822Z ioredis:redis write command[10.18.62.156:6379 (publisher)]: 0 -> readonly([])
2021-10-29 17:41:15.822 +00:00: 2021-10-29T17:41:15.822Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.822 +00:00: 2021-10-29T17:41:15.822Z ioredis:redis write command[10.18.62.156:6379 (publisher)]: 0 -> readonly([])
2021-10-29 17:41:15.823 +00:00: 2021-10-29T17:41:15.823Z ioredis:redis status[10.18.75.185:6379 (publisher)]: connect -> ready
2021-10-29 17:41:15.823 +00:00: 2021-10-29T17:41:15.823Z ioredis:connection set the connection name [publisher]
2021-10-29 17:41:15.823 +00:00: 2021-10-29T17:41:15.823Z ioredis:redis write command[10.18.75.185:6379 (publisher)]: 0 -> client([ 'setname', 'publisher' ])
2021-10-29 17:41:15.823 +00:00: 2021-10-29T17:41:15.823Z ioredis:connection set the connection to readonly mode
2021-10-29 17:41:15.824 +00:00: 2021-10-29T17:41:15.824Z ioredis:redis write command[10.18.75.185:6379 (publisher)]: 0 -> readonly([])
2021-10-29 17:41:15.824 +00:00: 2021-10-29T17:41:15.824Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.824 +00:00: 2021-10-29T17:41:15.824Z ioredis:redis write command[10.18.75.185:6379 (publisher)]: 0 -> readonly([])
2021-10-29 17:41:15.824 +00:00: 2021-10-29T17:41:15.824Z ioredis:redis status[10.18.85.21:6379 (publisher)]: connect -> ready
2021-10-29 17:41:15.825 +00:00: 2021-10-29T17:41:15.824Z ioredis:connection set the connection name [publisher]
2021-10-29 17:41:15.825 +00:00: 2021-10-29T17:41:15.825Z ioredis:redis write command[10.18.85.21:6379 (publisher)]: 0 -> client([ 'setname', 'publisher' ])
2021-10-29 17:41:15.825 +00:00: 2021-10-29T17:41:15.825Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.825 +00:00: 2021-10-29T17:41:15.825Z ioredis:redis write command[10.18.85.21:6379 (publisher)]: 0 -> cluster([ 'info' ])
2021-10-29 17:41:15.826 +00:00: 2021-10-29T17:41:15.826Z ioredis:redis status[10.18.75.185:6379 (cache)]: connecting -> connect
2021-10-29 17:41:15.826 +00:00: 2021-10-29T17:41:15.826Z ioredis:redis write command[10.18.75.185:6379 (cache)]: 0 -> info([])
2021-10-29 17:41:15.828 +00:00: 2021-10-29T17:41:15.828Z ioredis:redis status[10.18.62.156:6379 (cache)]: connecting -> connect
2021-10-29 17:41:15.828 +00:00: 2021-10-29T17:41:15.828Z ioredis:redis write command[10.18.62.156:6379 (cache)]: 0 -> info([])
2021-10-29 17:41:15.829 +00:00: 2021-10-29T17:41:15.829Z ioredis:redis status[10.18.85.21:6379 (cache)]: connecting -> connect
2021-10-29 17:41:15.830 +00:00: 2021-10-29T17:41:15.829Z ioredis:redis write command[10.18.85.21:6379 (cache)]: 0 -> info([])
2021-10-29 17:41:15.831 +00:00: 2021-10-29T17:41:15.831Z ioredis:redis status[10.18.75.185:6379 (subscriber)]: connecting -> connect
2021-10-29 17:41:15.831 +00:00: 2021-10-29T17:41:15.831Z ioredis:redis write command[10.18.75.185:6379 (subscriber)]: 0 -> info([])
2021-10-29 17:41:15.832 +00:00: 2021-10-29T17:41:15.832Z ioredis:cluster status: connect -> ready
2021-10-29 17:41:15.833 +00:00: 2021-10-29T17:41:15.833Z ioredis:cluster status: connect -> ready
2021-10-29 17:41:15.833 +00:00: 2021-10-29T17:41:15.833Z ioredis:redis status[10.18.75.185:6379 (cache)]: connect -> ready
2021-10-29 17:41:15.833 +00:00: 2021-10-29T17:41:15.833Z ioredis:connection set the connection name [cache]
2021-10-29 17:41:15.834 +00:00: 2021-10-29T17:41:15.834Z ioredis:redis write command[10.18.75.185:6379 (cache)]: 0 -> client([ 'setname', 'cache' ])
2021-10-29 17:41:15.834 +00:00: 2021-10-29T17:41:15.834Z ioredis:connection set the connection to readonly mode
2021-10-29 17:41:15.834 +00:00: 2021-10-29T17:41:15.834Z ioredis:redis write command[10.18.75.185:6379 (cache)]: 0 -> readonly([])
2021-10-29 17:41:15.834 +00:00: 2021-10-29T17:41:15.834Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.834 +00:00: 2021-10-29T17:41:15.834Z ioredis:redis write command[10.18.75.185:6379 (cache)]: 0 -> readonly([])
2021-10-29 17:41:15.835 +00:00: 2021-10-29T17:41:15.835Z ioredis:redis status[10.18.62.156:6379 (cache)]: connect -> ready
2021-10-29 17:41:15.835 +00:00: 2021-10-29T17:41:15.835Z ioredis:connection set the connection name [cache]
2021-10-29 17:41:15.835 +00:00: 2021-10-29T17:41:15.835Z ioredis:redis write command[10.18.62.156:6379 (cache)]: 0 -> client([ 'setname', 'cache' ])
2021-10-29 17:41:15.835 +00:00: 2021-10-29T17:41:15.835Z ioredis:connection set the connection to readonly mode
2021-10-29 17:41:15.835 +00:00: 2021-10-29T17:41:15.835Z ioredis:redis write command[10.18.62.156:6379 (cache)]: 0 -> readonly([])
2021-10-29 17:41:15.835 +00:00: 2021-10-29T17:41:15.835Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.835 +00:00: 2021-10-29T17:41:15.835Z ioredis:redis write command[10.18.62.156:6379 (cache)]: 0 -> readonly([])
2021-10-29 17:41:15.836 +00:00: 2021-10-29T17:41:15.836Z ioredis:redis status[10.18.85.21:6379 (cache)]: connect -> ready
2021-10-29 17:41:15.836 +00:00: 2021-10-29T17:41:15.836Z ioredis:connection set the connection name [cache]
2021-10-29 17:41:15.836 +00:00: 2021-10-29T17:41:15.836Z ioredis:redis write command[10.18.85.21:6379 (cache)]: 0 -> client([ 'setname', 'cache' ])
2021-10-29 17:41:15.836 +00:00: 2021-10-29T17:41:15.836Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.837 +00:00: 2021-10-29T17:41:15.837Z ioredis:redis write command[10.18.85.21:6379 (cache)]: 0 -> cluster([ 'info' ])
2021-10-29 17:41:15.838 +00:00: 2021-10-29T17:41:15.838Z ioredis:redis status[10.18.81.137:6379 (subscriber)]: connecting -> connect
2021-10-29 17:41:15.838 +00:00: 2021-10-29T17:41:15.838Z ioredis:redis write command[10.18.81.137:6379 (subscriber)]: 0 -> info([])
2021-10-29 17:41:15.839 +00:00: 2021-10-29T17:41:15.839Z ioredis:redis status[10.18.62.156:6379 (subscriber)]: connecting -> connect
2021-10-29 17:41:15.840 +00:00: 2021-10-29T17:41:15.840Z ioredis:redis write command[10.18.62.156:6379 (subscriber)]: 0 -> info([])
2021-10-29 17:41:15.840 +00:00: 2021-10-29T17:41:15.840Z ioredis:cluster status: connect -> ready
2021-10-29 17:41:15.841 +00:00: 2021-10-29T17:41:15.841Z ioredis:redis status[10.18.81.137:6379 (subscriber)]: connect -> ready
2021-10-29 17:41:15.841 +00:00: 2021-10-29T17:41:15.841Z ioredis:connection set the connection name [subscriber]
2021-10-29 17:41:15.841 +00:00: 2021-10-29T17:41:15.841Z ioredis:redis write command[10.18.81.137:6379 (subscriber)]: 0 -> client([ 'setname', 'subscriber' ])
2021-10-29 17:41:15.841 +00:00: 2021-10-29T17:41:15.841Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.842 +00:00: 2021-10-29T17:41:15.841Z ioredis:redis write command[10.18.81.137:6379 (subscriber)]: 0 -> cluster([ 'info' ])
2021-10-29 17:41:15.842 +00:00: 2021-10-29T17:41:15.842Z ioredis:redis status[10.18.75.185:6379 (subscriber)]: connect -> ready
2021-10-29 17:41:15.842 +00:00: 2021-10-29T17:41:15.842Z ioredis:connection set the connection name [subscriber]
2021-10-29 17:41:15.842 +00:00: 2021-10-29T17:41:15.842Z ioredis:redis write command[10.18.75.185:6379 (subscriber)]: 0 -> client([ 'setname', 'subscriber' ])
2021-10-29 17:41:15.843 +00:00: 2021-10-29T17:41:15.843Z ioredis:connection set the connection to readonly mode
2021-10-29 17:41:15.843 +00:00: 2021-10-29T17:41:15.843Z ioredis:redis write command[10.18.75.185:6379 (subscriber)]: 0 -> readonly([])
2021-10-29 17:41:15.843 +00:00: 2021-10-29T17:41:15.843Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.843 +00:00: 2021-10-29T17:41:15.843Z ioredis:redis write command[10.18.75.185:6379 (subscriber)]: 0 -> readonly([])
2021-10-29 17:41:15.843 +00:00: 2021-10-29T17:41:15.843Z ioredis:cluster status: connect -> ready
2021-10-29 17:41:15.844 +00:00: 2021-10-29T17:41:15.844Z ioredis:redis status[10.18.62.156:6379 (subscriber)]: connect -> ready
2021-10-29 17:41:15.844 +00:00: 2021-10-29T17:41:15.844Z ioredis:connection set the connection name [subscriber]
2021-10-29 17:41:15.844 +00:00: 2021-10-29T17:41:15.844Z ioredis:redis write command[10.18.62.156:6379 (subscriber)]: 0 -> client([ 'setname', 'subscriber' ])
2021-10-29 17:41:15.845 +00:00: 2021-10-29T17:41:15.845Z ioredis:connection set the connection to readonly mode
2021-10-29 17:41:15.845 +00:00: 2021-10-29T17:41:15.845Z ioredis:redis write command[10.18.62.156:6379 (subscriber)]: 0 -> readonly([])
2021-10-29 17:41:15.845 +00:00: 2021-10-29T17:41:15.845Z ioredis:connection send 1 commands in offline queue
2021-10-29 17:41:15.854 +00:00: 2021-10-29T17:41:15.845Z ioredis:redis write command[10.18.62.156:6379 (subscriber)]: 0 -> readonly([])

</details>

@Cellule
Copy link
Author

Cellule commented Nov 1, 2021

I enabled DEBUG=ioredis:* on a production server and got 1 reconnect occurrence
redis-reconnect.log

Out of the blue it seems the whole cluster decided to reconnect, I don't see any errors prior

| 2021-11-01 18:08:43.028 +00:00: 2021-11-01T18:08:43.028Z ioredis:cluster status: [empty] -> connecting                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| 2021-11-01 18:08:43.028 +00:00: 2021-11-01T18:08:43.028Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.028 +00:00: 2021-11-01T18:08:43.028Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.028 +00:00: 2021-11-01T18:08:43.028Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.028 +00:00: 2021-11-01T18:08:43.028Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.029 +00:00: 2021-11-01T18:08:43.029Z ioredis:cluster status: [empty] -> connecting                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| 2021-11-01 18:08:43.029 +00:00: 2021-11-01T18:08:43.029Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.029 +00:00: 2021-11-01T18:08:43.029Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.029 +00:00: 2021-11-01T18:08:43.029Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.029 +00:00: 2021-11-01T18:08:43.029Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.030 +00:00: 2021-11-01T18:08:43.030Z ioredis:cluster status: [empty] -> connecting                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| 2021-11-01 18:08:43.030 +00:00: 2021-11-01T18:08:43.030Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.030 +00:00: 2021-11-01T18:08:43.030Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.030 +00:00: 2021-11-01T18:08:43.030Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.030 +00:00: 2021-11-01T18:08:43.030Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.030 +00:00: 2021-11-01T18:08:43.030Z ioredis:cluster status: [empty] -> connecting                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| 2021-11-01 18:08:43.030 +00:00: 2021-11-01T18:08:43.030Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.030 +00:00: 2021-11-01T18:08:43.030Z ioredis:cluster resolved hostname mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0001-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.030 +00:00: 2021-11-01T18:08:43.030Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-001.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |
| 2021-11-01 18:08:43.030 +00:00: 2021-11-01T18:08:43.030Z ioredis:cluster resolved hostname mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com to IP mx-prod-app-redis-0002-002.mx-prod-app-redis.gzcra2.usw2.cache.amazonaws.com                                                                                                                                                                                                                                                                                                                                                                                                       |

@Cellule
Copy link
Author

Cellule commented Nov 1, 2021

Well that was a fun ride... turns out a few months ago one of our Devs started creating new redis cluster connection on a certain request...
The ioredis:* logs turned to be useful since I couldn't for the life of me see any reason ioredis would start a new cluster connection from scratch.

@Cellule Cellule closed this as completed Nov 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sikmir @Cellule