Support for sending OpenTelemetry Tracing over HTTP/Protobuf endpoint

[dceravigupta]

Title: Support for sending OpenTelemetry Tracing over HTTP/Protobuf endpoint

Description:

Today, Envoy support OpenTelemetry based tracing but only over a gRPC endpoint. Are there any plan to support for HTTP/Protobuf endpoint?

My use-case: I'm using Fluent Bit to log Envoy's access logs to my telemetry store but when it comes to OTel, Fluent Bit only supports listening to OTLP events over HTTP/Protobuf endpoint and not gRPC. Hence wondering are there any plans to add support for sending OTel traces over a HTTP/Protobuf endpoint along with gRPC endpoint?

[optional Relevant Links:]

Any extra documentation required to understand the issue.
https://docs.fluentbit.io/manual/pipeline/inputs/opentelemetry

[lizan]

@itamarkam @yanavlasov

[itamarkam]

Just to clarify, the title refers to Tracing, but the Fluent Bit story refers to Logging - @dceravigupta which one are you referring to (or is it both)?

[dceravigupta]

@itamarkam, right now I'm looking for Tracing but once I get that working I will also like to use Envoy OTel based access logs, so to answer your question, I was referring to kind both.

Also, on the Fluent Bit OTel plugin page, it says that it has stable support for both Traces and Logs over HTTP/Protobuf endpoint.

https://docs.fluentbit.io/manual/pipeline/inputs/opentelemetry#getting-started

[itamarkam]

I'm not familiar enough with Envoy Tracing (@AlexanderEllis is), but for Access Logs we used the existing Envoy grpc access logging infrastructure and tweaked it to send OTel. AFAICT there's no existing HTTP access logging in Envoy.
Maybe it's possible to reuse some of the batching and buffering logic in the grpc access loggers in an HTTP access logger but I don't know how simple that would be - @yanavlasov any thoughts?

[AlexanderEllis]

For tracing, there was already some discussion around adding an OTLP/HTTP implementation (#9958 (comment)) — I think this would be a good improvement to the tracer, and it shouldn't be to onerous to add. @esigo had mentioned some interest in pursuing this, but if they don't have the cycles we can always load balance to someone else that's interested 👍

[arminru]

There's an OTLP/HTTP exporter in OpenTelemetry C++ and it might be possible to borrow from that one for the Envoy implementation:
https://github.com/open-telemetry/opentelemetry-cpp/tree/main/exporters/otlp

[AlexanderEllis]

Took a stab at adding this, and I think it should be pretty doable to integrate with the existing tracer. The changes look roughly like the following:

• Refactor the OpenTelemetryGrpcTraceExporter to use a more general base class (I'm thinking OpenTelemetryTraceExporter) that the new HTTP exporter can also use
• Add new config to the OpenTelemetryConfig for the HTTP exporting — just adding a single HttpFormat option for now (binary protobuf), but we can add JSON protobuf in the future.
• Add the OpenTelemetryHttpTraceExporter, wire it up to config pipe through , and implement the log method to send spans

You can preview the very rough changes here, though they still need tests (and definitely a proper code review!)

So far I have a basic exporting setup (Envoy with a Nighthawk cluster, exporting to the OpenTelemetry collector) with the following config for the new HTTP exporting (note the 4318 port on the collector):

         tracing:
            provider:
              name: envoy.tracers.opentelemetry
              typed_config:
                "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
                service_name: "alexellis-foo"
                http_config:
                  http_uri:
                    uri: "https://www.example.com/v1/traces"
                    cluster: opentelemetry_collector
                    timeout: 0.250s
                  http_format: BINARY_PROTOBUF
                  collector_path: "/v1/traces"
                  collector_hostname: "example.com"
 clusters:
  - name: local_service
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: local_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 0.0.0.0
                port_value: 10000
  - name: opentelemetry_collector
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: opentelemetry_collector
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 0.0.0.0
                port_value: 4318

Looks like the spans came through to the collector (the static_delay: 0.5s latency in the Nighthawk config is immediately visible in the egress span, which is nice):

2023-06-20T00:24:33.757-0400	INFO	loggingexporter/logging_exporter.go:43	TracesExporter	{"#spans": 2}
2023-06-20T00:24:33.759-0400	DEBUG	loggingexporter/logging_exporter.go:52	ResourceSpans #0
Resource SchemaURL: 
Resource labels:
     -> service.name: STRING(alexellis-foo)
ScopeSpans #0
ScopeSpans SchemaURL: 
InstrumentationScope  
Span #0
    Trace ID       : 900142cdcbf19cdda0f1e91fbf39b397
    Parent ID      : c34e252f1314cf8f
    ID             : 7a0fd92335474c5c
    Name           : router local_service egress
    Kind           : SPAN_KIND_SERVER
    Start time     : 2023-06-20 04:24:31.765652 +0000 UTC
    End time       : 2023-06-20 04:24:32.274974 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message : 
Attributes:
     -> http.protocol: STRING(HTTP/1.1)
     -> upstream_address: STRING(0.0.0.0:10000)
     -> peer.address: STRING(0.0.0.0:10000)
     -> component: STRING(proxy)
     -> upstream_cluster: STRING(local_service)
     -> upstream_cluster.name: STRING(local_service)
     -> http.status_code: STRING(200)
     -> response_flags: STRING(-)
Span #1
    Trace ID       : 900142cdcbf19cdda0f1e91fbf39b397
    Parent ID      : 
    ID             : c34e252f1314cf8f
    Name           : ingress
    Kind           : SPAN_KIND_SERVER
    Start time     : 2023-06-20 04:24:31.763589 +0000 UTC
    End time       : 2023-06-20 04:24:32.275398 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message : 
Attributes:
     -> node_id: STRING()
     -> zone: STRING()
     -> guid:x-request-id: STRING(c99cac09-c78a-9006-aace-964ed903c734)
     -> http.url: STRING(http://localhost:10001/)
     -> http.method: STRING(GET)
     -> downstream_cluster: STRING(-)
     -> user_agent: STRING(curl/7.77.0)
     -> http.protocol: STRING(HTTP/1.1)
     -> peer.address: STRING(127.0.0.1)
     -> request_size: STRING(0)
     -> response_size: STRING(10)
     -> component: STRING(proxy)
     -> upstream_cluster: STRING(local_service)
     -> upstream_cluster.name: STRING(local_service)
     -> http.status_code: STRING(200)
     -> response_flags: STRING(-)

I still need to write some tests for it, but hopefully should have a PR in good shape pretty soon 👍

[joaopgrassi]

@AlexanderEllis I would be happy to collaborate and help with it however I can. Feel free to reach out here or tag me on the PR for reviews. One thing I missed in the changes you linked is (disregard if you already worked on it in the meantime) a way to configure HTTP headers. Several back-ends use the Authorization header as a way to send API tokens and the like to authenticate export request.

[joaopgrassi]

Hi @AlexanderEllis, sorry for nagging, but I'm interested in getting this feature in, so please let me know how I can help! If you don't have cycles to work on it, I'd be happy to take over. Let me know your thoughts. Thanks!

[AlexanderEllis]

Hey @joaopgrassi , no problem at all, thanks for keeping me honest! Sorry for the delay here — was away and busy for the last few weeks.

I wrapped up a bit of the metrics and testing, and I created a draft PR for us to look at (and to run all of the tests that I'll probably have to iterate on).

For the HTTP headers — I don't think I saw this in the other tracers, but I think something like that would make a lot of sense. Would that be best placed in the upstream_http_protocol_options for the OpenTelemetry collector cluster? E.g. in a header mutation append config.

I must admit I'm very rusty on my Envoy best practices, so I'd definitely defer to the experts on this one.

[joaopgrassi]

Hi @AlexanderEllis ! No problem!

Awesome, I will take a look at the PR!

About the HTTP headers, I'm a beginner in Envoy but I looked at some of the other "tracers" available, and https://github.com/envoyproxy/envoy/blob/main/source/extensions/tracers/datadog/agent_http_client.cc#L54 this one also sends via HTTP and it seems configures HTTP headers. I did not find yet how those are passed down/configured though.