Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[OpenTelemetry] Support OpenTelemetry Span Events #4715

Closed
cyrille-leclerc opened this issue Feb 10, 2021 · 7 comments · Fixed by #6122
Closed

[OpenTelemetry] Support OpenTelemetry Span Events #4715

cyrille-leclerc opened this issue Feb 10, 2021 · 7 comments · Fixed by #6122

Comments

@cyrille-leclerc
Copy link
Contributor

cyrille-leclerc commented Feb 10, 2021

Elastic APM silently ignores the Otel Span Events.

Specification of OpenTelemetry Span Events: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#add-events

Example of Otel Span Event we want to support:

span.addEvent("order-creation", Attributes.of(
        EcommerceAppOpenTelemetryAttributes.CUSTOMER_ID, customerId,
        EcommerceAppOpenTelemetryAttributes.ORDER_PRICE, orderPrice,
        EcommerceAppOpenTelemetryAttributes.PAYMENT_METHOD, paymentMethod,
        EcommerceAppOpenTelemetryAttributes.SHIPPING_METHOD, shippingMethod,
        EcommerceAppOpenTelemetryAttributes.SHIPPING_COUNTRY, shippingCountry));

https://github.com/cyrille-leclerc/my-shopping-cart/blob/08f33761010d268793edb6e1d2101eaa579d830f/frontend-java/src/main/java/com/mycompany/ecommerce/controller/OrderController.java#L96

Source code ignoring the span events: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/91cbf1fd7f4eeea0087fdf782e3ff5b04e3ac90f/exporter/elasticexporter/internal/translator/elastic/traces.go#L88

@paulgrav
Copy link

Would be good get some movement on this issue. Currently our devs are logging to stdout given that Span Events aren’t supported in either the Elastic Exporter or Elastic APM Server, or worse bunging stuff in attributes.

@axw
Copy link
Member

axw commented Aug 10, 2021

@paulgrav apologies for the silence, we've been rather busy. This isn't currently scheduled, but I'm hoping we'll be able to get to it within the next few months. My current thinking is that we'll index span events as logs, such that they'll show up in the trace logs UI:

image

Please let me know if that sounds off.

One of reasons why we haven't jumped on this is that APM Server does not create any log indices. We are working on migrating APM to data streams, and as part of that work we'll be introducing logs support. To get span events in sooner, we might support them only when producing data streams. This would mean using the Fleet integration, or waiting for #5914.

@paulgrav
Copy link

Thanks for the update.

I think indexing them in logs makes sense. That’s all the span events are. Our current workaround is to not call the Span Event API and send the data to stdout.

The workaround itself isn’t onerous, it’s a small code change. We have multiple services using the Jaeger tracer, traces are stored in ES, and are queryable in the Jaeger UI. Span logs, as they are known in the Open Tracing context, just work! We are trying to encourage developers to instead send their traces to Open Telemetry, which then forwards the traces to Elastic APM, but I then need to tell them that all their Span Log data will be dropped 😬

a) it’s not a good story for Open Telemetry or Elastic APM. It’s makes it harder for me to evangelise using those technologies.
b) the work around requires code changes that will become redundant once this issue is resolved

The likelihood is teams will be unwilling to migrate to Otel+ElasticAPM if their current tracing setup works, and they have to add issues to their backlog to work around the lack of Span Events.

Thinking about it, it’s probably still worth getting devs to send their data to Otel, but then have Otel forward the traces to Jaeger. Then once Span Events are supported by ElasticAPM, we can change the config to forward the traces to ElasticAPM.

@psk001
Copy link

psk001 commented Sep 14, 2022

Hi I have come across this issue while using openTelemetry with elastic APM with a distributed nodeJs service. My span events are stored in elastic indices and there data streams as well. But they are still not visible in logs part of traces as mentioned by @axw above. I just want to know if there is support for this feature or is there any issue with my apm server settings?

@axw
Copy link
Member

axw commented Sep 14, 2022

@psk001 yes, span events are supported. Please open a topic in Discuss. The more information you can provide the better: which versions you're using, and ideally a sample program.

@psk001
Copy link

psk001 commented Sep 14, 2022

thanks for the quick reply @axw , I'll check on Discuss. Cheers

@astorm
Copy link
Contributor

astorm commented Sep 14, 2022

Looks like the discuss issue was opened here: https://discuss.elastic.co/t/opentelemetry-span-events-are-not-visible-in-apm-ui/314419

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants