Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retrying not working when executing Redis Pipeline for retry_on_error exceptions #2973

Closed
jfcalcerrada opened this issue Oct 2, 2023 · 2 comments · Fixed by #3134
Closed

Comments

@jfcalcerrada
Copy link

Version: 4.3.6 to 5.0.1

Platform: 3.8 to 3.11

Description: redis-py does not attempt to reconnect when executing a pipeline for the errors provided in retry_on_error, instead the exceptions are bubbled up.

This library accepts a retry policy and a list of Errors/Exceptions to attempt a retry.

When executing a single command in the Sync and Async versions, it calls call_with_retry passing _disconnect_raise as a callback in the client.py and asyncio/client.py. In both cases, _disconnect_raise won't bubble up exceptions in the retry_on_error list. See:

As exceptions aren't bubbled up, retry will work as expected when calling that callback:

fail(error)

But this is not the case for Pipeline on both versions (Sync and Async). When executing a pipeline, this time the _disconnect_reset_raise callback is passed in call_with_retry:

def _disconnect_raise_reset(self, conn: Redis, error: Exception) -> None:

This callback DO bubble up all exceptions but TimeError, as the exception is bubbled up, call_with_retry will raise an exception when executing this line of code:

fail(error)

Thus bubbling up the exception all the way up, ignoring the retry_on_error list and the retry policy in place.

@jfcalcerrada
Copy link
Author

This is the same issue reported here: #2179

But in this case is affecting client.Pipeline and asyncio.client.Pipeline

@cknv
Copy link

cknv commented Nov 14, 2023

Unless I am mistaken the same is also true for the PubSub clients, both sync and async.

Bringing the different versions in sync is probably not going to be too difficult, if there is interest from the maintainers.

w-miller added a commit to pexip/os-python-redis that referenced this issue Feb 2, 2024
Extend the fix from bea72995fd39b01e2f0a1682b16b6c7690933f36 to apply to
pipeline and pubsub as well.

Fixes redis/redis-py#2973
w-miller added a commit to w-miller/redis-py that referenced this issue Feb 2, 2024
Extend the fix from bea7299 to apply to
pipeline and pubsub as well.

Fixes redis#2973
dvora-h added a commit that referenced this issue Feb 19, 2024
* Fix retry logic for pubsub and pipeline

Extend the fix from bea7299 to apply to
pipeline and pubsub as well.

Fixes #2973

* fix isort

---------

Co-authored-by: dvora-h <[email protected]>
dvora-h added a commit that referenced this issue Feb 25, 2024
* Fix retry logic for pubsub and pipeline

Extend the fix from bea7299 to apply to
pipeline and pubsub as well.

Fixes #2973

* fix isort

---------

Co-authored-by: dvora-h <[email protected]>
dvora-h added a commit that referenced this issue Feb 26, 2024
* Bump actions/checkout from 3 to 4 (#2969)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump rojopolis/spellcheck-github-actions from 0.33.1 to 0.34.0 (#2970)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix type hint (#2963)

Co-authored-by: d184230 <[email protected]>

* Don't perform blocking connect inside the BlockingConnectionQueue Condition variable. (#2997)

* Creating CODEOWNERS for the examples (#2993)

* Close various objects created during asyncio tests (#3005)

* Close various objects created during asyncio tests

* Fix resource leake in test_cwe_404.py
Need to wait for individual handler tasks when shutting down server.

* Linking to Redis resources (#3006)

* Add GEOSHAPE field type for index creation of RediSearch (#2957)

* first pass of geoshape index type

* first attempt at test, but demonstrates the initial commit is broken

* fix new field + fix tests

* work on linter

* more linter

* try to mark test with correct fixture

* fix linter

* Better deal with "lost" connections for async Redis (#2999)

* Allow tracking/reporting and closing of "lost" connections.
ConnectionPool keeps a WeakSet of in_use connections, allowing lost ones to be collected.
Collection produces a warning and closes the underlying transport.

* Add tests for the __del__ handlers of async Redis and Connection objects

* capture expected warnings in the test

* lint

* Update client.py sleep_time typing for run_in_thread function (#2977)

Changed from 
`sleep_time: int = 0`
to
`sleep_time: float = 0.0`
To avoid Pylance complaining: 
`Argument of type "float" cannot be assigned to parameter "sleep_time" of type "int" in function "run_in_thread"
  "float" is incompatible with "int"`

* Fix BlockingConnectionPool.from_url parsing of timeout in query args #2983 (#2984)

Co-authored-by: Romain Fliedel <[email protected]>

* Fix parsing resp3 dicts (#2982)

* Update ocsp.py (#3022)

* Bump rojopolis/spellcheck-github-actions from 0.34.0 to 0.35.0 (#3060)

Bumps [rojopolis/spellcheck-github-actions](https://github.com/rojopolis/spellcheck-github-actions) from 0.34.0 to 0.35.0.
- [Release notes](https://github.com/rojopolis/spellcheck-github-actions/releases)
- [Changelog](https://github.com/rojopolis/spellcheck-github-actions/blob/master/CHANGELOG.md)
- [Commits](rojopolis/spellcheck-github-actions@0.34.0...0.35.0)

---
updated-dependencies:
- dependency-name: rojopolis/spellcheck-github-actions
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Use `disable_decoding` in async `read_response`. (#3042)

* Add "sum" to DUPLICATE_POLICY documentation of TS.CREATE, TS.ADD and TS.ALTER (#3027)

* Update advanced_features.rst (#3019)

* Fix typos. (#3016)

* Fix parsing of `FT.PROFILE` result (#3063)

* Fix parsing of ft.profile result

* test

* Make the connection callback methods public again, add documentation (#2980)

* Fix reported version of deprecations in asyncio.client (#2968)

* Allow the parsing of the asking command to forward original options (#3012)

Co-authored-by: dvora-h <[email protected]>

* Fix Specifying Target Nodes broken hyperlink (#3072)

The typo cause hyperlinks to fail.

* Fix return types in json commands (#3071)

* Fix return types in JSONCommands class

* Update CHANGES

* fix acl_genpass with bits (#3062)

* Bump github/codeql-action from 2 to 3 (#3096)

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@v2...v3)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/upload-artifact from 3 to 4 (#3097)

Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v3...v4)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/setup-python from 4 to 5 (#3095)

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Always sending codecov (#3101)

* filter commits for main branch (#3036)

* fix(docs): organize cluster mode part of lua scripting (#3073)

* Fix typing for `HashCommand.hdel` (#3029)

Co-authored-by: dvora-h <[email protected]>

* Adding lock_name to LockError (#3023)

* Adding lock_name to LockError

* Adding lock_name to LockError

---------

Co-authored-by: dvora-h <[email protected]>

* Fix objlen type hint (#2966)

* Fix objlen type hint

* Update redis/commands/json/commands.py

Co-authored-by: dvora-h <[email protected]>

* linters

---------

Co-authored-by: dvora-h <[email protected]>

* Fix type hint of arbitrary argument lists (#2908)

* Fix: hset unexpectedly mutates the list passed to items (#3103)

* Fix possible pipeline connections leak (#3104)

* Update cluster.py

When Executing "n.write()" may generate some unknown errors(e.g. DataError), which could result in the connection not being released.

* Update cluster.py

* Update cluster.py

release connection move to "try...finally"

* Update cluster.py

 fix the linters

* fix problems of code review

* Add modules support to async RedisCluster (#3115)

* Fix grammer in BlockingConnectionPool class documentation (#3120)

Co-authored-by: ahmedabdou14 <root@xps>

* release already acquired connections on ClusterPipeline, when get_connection raises an exception (#3133)

Signed-off-by: zach.lee <[email protected]>

* Bump actions/stale from 3 to 9 (#3132)

Bumps [actions/stale](https://github.com/actions/stale) from 3 to 9.
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](actions/stale@v3...v9)

---
updated-dependencies:
- dependency-name: actions/stale
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump codecov/codecov-action from 3 to 4 (#3131)

Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 3 to 4.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@v3...v4)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Allow to control the minimum SSL version (#3127)

* Allow to control the minimum SSL version

It's useful for applications that has strict security requirements.

* Add tests for minimum SSL version

The commit updates test_tcp_ssl_connect for both sync and async
connections. Now it sets the minimum SSL version. The test is ran with
both TLSv1.2 and TLSv1.3 (if supported).

A new test case is test_tcp_ssl_version_mismatch. The test added for
both sync and async connections. It uses TLS 1.3 on the client side,
and TLS 1.2 on the server side. It expects a connection error. The
test is skipped if TLS 1.3 is not supported.

* Add example of using a minimum TLS version

* docs: Add timeout parameter for get_message example (#3129)

The `get_message()` method in asyncio PubSub has a `timeout` argument that defaults to 0.0, causing it to immediately return. This can cause high CPU usage with the given code example and should not be recommended. By setting `timeout=None`, it works with much more efficient resource usage.

* Revert stale isuue version update (#3142)

* Update connection.py (#3149)

Exception ignored in: <function Redis.__del__ at ...>
Traceback ....
TypeError: 'NoneType' object cannot be interpreted as an integer.

This happens when closing the connection within a spawned Process (multiprocess).

* Fix retry logic for pubsub and pipeline (#3134)

* Fix retry logic for pubsub and pipeline

Extend the fix from bea7299 to apply to
pipeline and pubsub as well.

Fixes #2973

* fix isort

---------

Co-authored-by: dvora-h <[email protected]>

* Update install_requires (#3109)

---------

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: zach.lee <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Artem Diubkov <[email protected]>
Co-authored-by: d184230 <[email protected]>
Co-authored-by: Kristján Valur Jónsson <[email protected]>
Co-authored-by: Chayim <[email protected]>
Co-authored-by: Shaya Potter <[email protected]>
Co-authored-by: Bosheng (Daniel) Zhang <[email protected]>
Co-authored-by: Romain <[email protected]>
Co-authored-by: Romain Fliedel <[email protected]>
Co-authored-by: Aniket Patil <[email protected]>
Co-authored-by: Stanisław Denkowski <[email protected]>
Co-authored-by: Pedram Parsian <[email protected]>
Co-authored-by: BackflipPenguin <[email protected]>
Co-authored-by: AYMEN Mohammed <[email protected]>
Co-authored-by: Zachary Ware <[email protected]>
Co-authored-by: Tyler Bream (Event pipeline) <[email protected]>
Co-authored-by: Binbin <[email protected]>
Co-authored-by: Pármenas Haniel <[email protected]>
Co-authored-by: Wei-Hsiang (Matt) Wang <[email protected]>
Co-authored-by: Dmitry Kulazhenko <[email protected]>
Co-authored-by: Dan Lousqui <[email protected]>
Co-authored-by: trkwyk <[email protected]>
Co-authored-by: SwordHeart <[email protected]>
Co-authored-by: Jason <[email protected]>
Co-authored-by: Ahmed Ashraf <[email protected]>
Co-authored-by: ahmedabdou14 <root@xps>
Co-authored-by: Dongkeun Lee <[email protected]>
Co-authored-by: poiuj <[email protected]>
Co-authored-by: Qiangning Hong <[email protected]>
Co-authored-by: wKollendorf <[email protected]>
Co-authored-by: Will Miller <[email protected]>
vladvildanov pushed a commit that referenced this issue Sep 27, 2024
* Fix retry logic for pubsub and pipeline

Extend the fix from bea7299 to apply to
pipeline and pubsub as well.

Fixes #2973

* fix isort

---------

Co-authored-by: dvora-h <[email protected]>
vladvildanov pushed a commit that referenced this issue Sep 27, 2024
* Fix retry logic for pubsub and pipeline

Extend the fix from bea7299 to apply to
pipeline and pubsub as well.

Fixes #2973

* fix isort

---------

Co-authored-by: dvora-h <[email protected]>
vladvildanov pushed a commit that referenced this issue Sep 27, 2024
* Fix retry logic for pubsub and pipeline

Extend the fix from bea7299 to apply to
pipeline and pubsub as well.

Fixes #2973

* fix isort

---------

Co-authored-by: dvora-h <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants