Make reds subscription more resilient

[bodymindarts]

Any time we are subscribing to redis messages - we need an outer loop that reconnects the subscriber if something fails.
Ideally handled in our Subscriber client and not in calling code.
-> implies returning an mpsc or something like that.

First investgate expected behaviour of client library.

[bodymindarts]

First investgate expected behaviour of client library.

after looking closer it should already be re-connecting. Perhaps the custom-recconect-errors feature is missing - but not sure what is going on.

[enigbe]

The client library reconnects fine. I tried to recreate an environment similar to the one we have in staging locally. It is not quite exact but enough to test how the library behaves when there is a network partition. The challenge here may be linked to the sentinel going down during the upgrade.

[bodymindarts]

Upstream already has tickets for exactly this issue with fixes in flight:

• Connection management issues aembke/fred.rs#62
• Unable to reconnect to Redis Cluster in Kubernetes after cluster rollout aembke/fred.rs#56
• Incorrect Cluster Failover Behavior aembke/fred.rs#59