Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle redis.exceptions.WatchError as a non-error event in instrumentation #2668

Merged
merged 26 commits into from
Jul 15, 2024

Conversation

zhihali
Copy link
Contributor

@zhihali zhihali commented Jul 5, 2024

Description

Fixes # (issue)
When we face the WatchError when running redis instrumentation, Status Code: The status_code should be "UNSET" instead of "ERROR",
#2639

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

py312-test-instrumentation-redis

Does This PR Require a Core Repo Change?

  • Yes. - Link to PR:
  • No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added
  • Documentation has been updated

@zhihali zhihali requested a review from a team July 5, 2024 22:43
@zhihali zhihali changed the title Handle redis.exceptions.WatchError as a non-error event in instrumentation [wip]Handle redis.exceptions.WatchError as a non-error event in instrumentation Jul 6, 2024
@zhihali zhihali changed the title [wip]Handle redis.exceptions.WatchError as a non-error event in instrumentation Handle redis.exceptions.WatchError as a non-error event in instrumentation Jul 6, 2024
tox.ini Outdated Show resolved Hide resolved
zhihali and others added 2 commits July 10, 2024 12:19
remove fakeredis from tox.ini
better code format
@zhihali zhihali requested review from xrmx and emdneto July 10, 2024 11:39
Copy link
Member

@emdneto emdneto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we missing a test to check if any other error will have the StatusCode.ERROR?

@zhihali
Copy link
Contributor Author

zhihali commented Jul 10, 2024

Are we missing a test to check if any other error will have the StatusCode.ERROR?

Yes, and updated. Thanks for the review

@zhihali zhihali requested review from xrmx and emdneto July 11, 2024 16:57
@lzchen lzchen merged commit 7567efa into open-telemetry:main Jul 15, 2024
379 checks passed
@chrisguidry
Copy link
Contributor

Hi folks, shouldn't the WatchError be re-raised? It's part of a redis flow control mechanism but this change will have the instrumentation swallowing the exception.

Comment on lines +403 to +412
try:
redis_client = FakeRedis()
async with redis_client.pipeline(transaction=False) as pipe:
await pipe.watch("a")
await redis_client.set("a", "bad")
pipe.multi()
await pipe.set("a", "1")
await pipe.execute()
except WatchError:
pass
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd recommend this instead, so that we ensure the WatchError is still raised

Suggested change
try:
redis_client = FakeRedis()
async with redis_client.pipeline(transaction=False) as pipe:
await pipe.watch("a")
await redis_client.set("a", "bad")
pipe.multi()
await pipe.set("a", "1")
await pipe.execute()
except WatchError:
pass
with pytest.raises(WatchError):
redis_client = FakeRedis()
async with redis_client.pipeline(transaction=False) as pipe:
await pipe.watch("a")
await redis_client.set("a", "bad")
pipe.multi()
await pipe.set("a", "1")
await pipe.execute()

Comment on lines +362 to +371
try:
redis_client = fakeredis.FakeStrictRedis()
pipe = redis_client.pipeline(transaction=True)
pipe.watch("a")
redis_client.set("a", "bad") # This will cause the WatchError
pipe.multi()
pipe.set("a", "1")
pipe.execute()
except WatchError:
pass
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
try:
redis_client = fakeredis.FakeStrictRedis()
pipe = redis_client.pipeline(transaction=True)
pipe.watch("a")
redis_client.set("a", "bad") # This will cause the WatchError
pipe.multi()
pipe.set("a", "1")
pipe.execute()
except WatchError:
pass
with pytest.raises(WatchError):
redis_client = fakeredis.FakeStrictRedis()
pipe = redis_client.pipeline(transaction=True)
pipe.watch("a")
redis_client.set("a", "bad") # This will cause the WatchError
pipe.multi()
pipe.set("a", "1")
pipe.execute()

@xrmx
Copy link
Contributor

xrmx commented Jul 17, 2024

@chrisguidry please open a new PR with the fix, you're commenting a PR already merged

chrisguidry added a commit to chrisguidry/opentelemetry-python-contrib that referenced this pull request Jul 17, 2024
In open-telemetry#2668, we started to avoid having the `redis.WatchError` mark the span as
having failed, but we were inadvertently suppressing that exception from the
calling code.  `redis.WatchError` is used for flow-of-control concurrency in
many applications, and applications depend on catching the error to handle
concurrent changes to keys during redis pipelines.

This re-raises the `WatchError` to keep the instrumentation transparent to the
application.

Fixes open-telemetry#2639
chrisguidry added a commit to chrisguidry/opentelemetry-python-contrib that referenced this pull request Jul 18, 2024
In open-telemetry#2668, we started to avoid having the `redis.WatchError` mark the span as
having failed, but we were inadvertently suppressing that exception from the
calling code.  `redis.WatchError` is used for flow-of-control concurrency in
many applications, and applications depend on catching the error to handle
concurrent changes to keys during redis pipelines.

This re-raises the `WatchError` to keep the instrumentation transparent to the
application.

Fixes open-telemetry#2639
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants