-
Notifications
You must be signed in to change notification settings - Fork 104
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spans aren't being marked as errors in Cloud Trace #730
Comments
This seems suspicious... opentelemetry-operations-go/exporter/trace/trace_proto.go Lines 162 to 163 in 6b51e22
|
Seems potentially related to #143 |
@aabmass do you remember why we set |
Unknown represents an unknown error, along the lines of HTTP 500 status code. Since OTel only has two possible statuses (OK and ERROR), gRPCs UNKNOWN (error) seems reasonable. Do you know what status codes actually show red in Cloud Trace? |
It does seem like we are doing the right thing based on https://pkg.go.dev/google.golang.org/genproto/googleapis/rpc/code#Code
I'll see if I can find the answer to that question. |
I tested all status codes, and none appear to make the span look like an error |
I'll reach out to the trace UI team. |
For further context, the way we're getting around this currently is by setting the attribute /http/status_code to 500 regardless of whether the context is for a HTTP server or not. It seems to be the only way to make the trace UI render it as an error. |
Hi @dashpole, is there any update on this? |
The cloud trace folks are aware of the issue, and suggested the same workaround pointed out above: #730 (comment). I'm not sure about timelines, but i'll post here when there are updates. |
+1. We are facing this problem as well. Thanks for pointing to the workaround! |
Lowering to p3 since the workaround is sufficient |
workaround is odd. adding http status code to code just so Google Trace can recognise, is not good. for example, if there is custom span (e.g. span_kind worker or consumer in open telemetry lingo) then Google Trace is not helpful. At least being able to configure what user threats as error would help. For example, in Grafana this is possible.
I spent some time to find where the error happen buy looking at the span attributes and finding grpc status 7. no colors at all, all blue. very hard to read. Google Trace needs to improve. finding error traces, is very important user feature. I hope you guys at Google realise that if this does not work, this is 👎🏻 for Google Trace Explorer product. |
Hi,
I'm instrumenting a gRPC server with OpenTelemetry and Google Cloud Trace. I can see spans in my Trace dashboard but they aren't being coloured red if an rpc returns an internal error. I'm using the
otelgrpc.UnaryServerInterceptor()
(code here) which callsspan.SetStatus
with the otel error code and the grpc message if any of these statuses are returnedgrpc_codes.Unknown
grpc_codes.DeadlineExceeded
grpc_codes.Unimplemented
grpc_codes.Internal
grpc_codes.Unavailable
grpc_codes.DataLoss
I've also tried calling
span.SetStatus
outside the interceptors and Cloud Trace doesn't colour them red either so I don't think the problem is with the interceptor code.I've created a simple demo app to reproduce this using the example grpc-go Greeter service with the addition of tracing using otel and cloudtrace.
When forcing a request to fail this is what I see in cloud trace
The interceptor has added the attribute
rpc.grpc.status_code: 13
but the span status isn't showing up.Ideally this would produce a red dot in the trace graph and the span would be coloured red.
Many thanks,
Andy
The text was updated successfully, but these errors were encountered: