Cloud Trace Propagator Doesn't Work with gRPC

[dgildeh]

The Cloud Trace Propagator uses the X-Cloud-Trace-Context header key, which has upper case letters. When using it with the gRPC instrumentation, it sets this as gRPC metadata which will throw a validation error and abort the request because of the upper case letters.

Please make the key all lower case, I tested it on my own services on Cloud Run using a Flask app (http from Google Cloud's Load Balancer which sets the x-cloud-trace-context header initially), and is passed on to my backend service using gRPC and it works with the metadata key set to the same, and appears that the AppServer component on Cloud Run in-between the two services also picks up the trace ID with the lowercase metadata header, and everything links up correctly as expected so I can now see my full end-to-end request on Google Trace correctly.

[aabmass]

Thanks for filing an issue. From what I can tell, gRPC metadata is case insensitive and gRPC should handle that. What is the actual validation error you are seeing?

[aabmass]

I think I will change it to lowercase here, and discuss with the opentelemetry python community. Ideally the gRPC instrumentation would handle propagators that used mixed case.

[dgildeh]

Great thanks! I didn't record the error but it was raised from the C libraries for gRPC and related to this issue I found I believe: grpc/grpc#9863

I've tested it and it appears to work lowercase for Google's services as well as gRPC and HTTP - you can find all my code that I've been testing with here which I just open sourced - I will remove my copy of the propagator as soon as this issue is fixed!

https://github.com/dgildeh/otel-python-cloud-run
Class: https://github.com/dgildeh/otel-python-cloud-run/blob/main/common-lib/common_lib/observe/cloud_propagator.py

[paralelocslucasmagalhaes]

Hi guys.. I tested the version and the problem still happening..

On my tests I notice the pattern of x-cloud-trace-context the Google's Services send was different that "trace_id/span_id;o=Flag". On my test I've received just "trace_id/span_id" without the ";o=Flag".

So.. when the code does "match = re.fullmatch(_TRACE_CONTEXT_HEADER_RE, header)" The re expression wasn't fullfiled and return None.

I have a suspicion that the change could be part of HTTP2.0 setup, cause others "REST Service" running on HTTP1.0 the x-cloud-trace-context pattern is the way we expected.

[aabmass]

@paralelocslucasmagalhaes this sounds like a different problem. Would you mind opening another issue in this repo with some more details?