Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow ignoring invalid SpanContext in OpenTelemetry TracingInterceptor #1174

Closed
cdavis-joy opened this issue Jul 26, 2023 · 2 comments · Fixed by #1178
Closed

Allow ignoring invalid SpanContext in OpenTelemetry TracingInterceptor #1174

cdavis-joy opened this issue Jul 26, 2023 · 2 comments · Fixed by #1178
Labels
enhancement New feature or request

Comments

@cdavis-joy
Copy link
Contributor

cdavis-joy commented Jul 26, 2023

Is your feature request related to a problem? Please describe.
We recently moved from OpenTracing to OpenTelemetry. Upon deploying, we found a long running workflow failed because it was trying to start a span from an OpenTracing parent span, so an error was returned here: https://github.com/temporalio/sdk-go/blob/master/contrib/opentelemetry/tracing_interceptor.go#L137

Describe the solution you'd like
Failures to trace should have the option to do so silently. Adding an option to the TracingInterceptor to ignore the invalid SpanContext would allow us to keep this long running workflow alive when the new version of the worker deploys. It's acceptable for spans to not connect to their parents for a short time; it is not acceptable for a critical workflow to fail because of a failure to trace.

Describe alternatives you've considered
The DataDog interceptor seems to handle this gracefully already. OpenTelemetry does not return any error when getting a SpanContext.

Additional context
I've added a commit on my fork that attempts to address this. I feel like this is likely a common thing that people might run into; running workflows that straddle two tracing implementations.

@cdavis-joy cdavis-joy added the enhancement New feature or request label Jul 26, 2023
@cdavis-joy
Copy link
Contributor Author

To zoom out a bit, the SDK is enforcing tracing to operate without error, and if it does not, workflows/activities cannot run. (see https://github.com/temporalio/sdk-go/blob/master/interceptor/tracing_interceptor.go#L375)

Maybe I'm totally off base here in thinking that workflows/activities should not function if tracing is not functioning, but I'm happy to hear any other thoughts.

@Quinn-With-Two-Ns
Copy link
Contributor

Yeah I think adding an option to ignore errors this case seems very reasonable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants