[GROW-3361] always release, instead of disconnect, when error occurs during get_connection

[zach-iee]

Pull Request check-list

Please make sure to review and check all of these items:

• Does $ tox pass with this change (including linting)?
• Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
• Is the new or changed code fully tested?
• Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
• Is there an example added to the examples folder (if applicable)?
• Was the change added to CHANGES file?

NOTE: these things are not required to open a PR and can be done
afterwards / while the PR is open.

Description of change

gevent.Timeout used to cause a connection leak, but that was patched with #8.

Now, i want to make sure that connections are not disconnected so often, so this PR makes any error from get_connection to always release a connection without disconnection. Disconnection is only necessary, when a connection is reading or writing, but get_connection is neither of those

additional changes:

• isinstance(e, self.retry._supported_errors) is lengthy, so i've also created a is_supported_error function to shorten the code
• add a static backoff, when we need to disconnect connections due to unforeseen exceptions
  • backoff from retry is not used, because it's not a retriable exception
  • similar to https://github.com/redis/redis-py/blob/6968431983c20a6f874a97d92c4d11366721b83b/redis/cluster.py#L1198

[codecov-commenter]

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.03% ⚠️

Comparison is base (ae88892) 92.37% compared to head (fa3864f) 92.35%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #11      +/-   ##
==========================================
- Coverage   92.37%   92.35%   -0.03%     
==========================================
  Files         119      119              
  Lines       30638    30678      +40     
==========================================
+ Hits        28302    28332      +30     
- Misses       2336     2346      +10     

Files Changed	Coverage Δ
redis/cluster.py	90.89% <100.00%> (+0.03%)	⬆️
redis/retry.py	100.00% <100.00%> (ø)
tests/test_cluster.py	96.90% <100.00%> (-0.16%)	⬇️
tests/test_retry.py	100.00% <100.00%> (ø)

... and 3 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[neoatlan]

Just for curious, Is there any reason for delaying after the disconnect the connections?

[zach-iee]

this code is a safety feature to catch all connection leaks, but it comes with a cost of disconnecting all connections used in this function. Ideally, this code block shouldn't be executed, but if executed due to an unknown error, it means that there probably is a problem with Soda or Redis. Reconnecting can put more load on a Redis node, so i thought static backoff could alleviate the load on Redis CPU