APM-Server and Envoy (or Istio) for tracing integration or driver.

[nerddelphi]

Is there any idea to integrate Istio Mixer or Envoy adapters with Elastic APM for tracing? I'd like using Istio and Elastic APM for a complete observability suite.

Envoy opentracing supported adapters are: Zipking, Jaeger and LighStep.

https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/observability/tracing#arch-overview-tracing

https://istio.io/docs/tasks/telemetry/distributed-tracing/
https://istio.io/faq/distributed-tracing/#how-to-support-tracing

[axw]

We've done some investigation into this, but nothing to show for it yet. Below is a bit of information on how I see this working in the future.

Envoy recently gained support for OpenCensus (envoyproxy/envoy#2456), including support for W3C Trace-Context. We could modify the APM Server to receive OpenCensus data from Envoy by implementing an OpenCensus Service receiver, translating the trace data to our schema. Once we've implemented elastic/apm#71 in the agents, then they would be able to extend the Envoy/OpenCensus traces as long as Envoy is configured to use W3C Trace-Context.

[nerddelphi]

We've done some investigation into this, but nothing to show for it yet. Below is a bit of information on how I see this working in the future.

Envoy recently gained support for OpenCensus (envoyproxy/envoy#2456), including support for W3C Trace-Context. We could modify the APM Server to receive OpenCensus data from Envoy by implementing an OpenCensus Service receiver, translating the trace data to our schema. Once we've implemented elastic/apm#71 in the agents, then they would be able to extend the Envoy/OpenCensus traces as long as Envoy is configured to use W3C Trace-Context.

Ok, @axw. It'll look great seeing that integration using OpenCensus Service receivers. I'm looking forward, then. Is there any roadmap or anything else? Anyway, thank you for the answer.

I'm talking in Envoy Slack channel and they told me that: https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/trace/v2/trace.proto?origin_team=T5CALEVRN#envoy-api-msg-config-trace-v2-dynamicotconfig -> "DynamicOtConfig is used to dynamically load a tracer from a shared library that implements the OpenTracing dynamic loading API."

[axw]

@nerddelphi this is not currently on the roadmap, but input like yours will help get it there :)

Regarding dynamically loading an OpenTracing library: this was how I previously thought we might do it, but it requires a C++ implementation which we currently do not have. It's still an option, but I suspect a lot more work. Also, OpenTracing is being deprecated in favour of OpenTelemetry.

[axw]

I looked a little closer at the Envoy OpenCensus implementation, and noticed that there's currently two exporters: Zipkin and StackDriver.

We already have an issue for implementing an OpenTelemetry exporter, and OpenTelemetry Service (WIP) has support for receiving traces in Zipkin format: https://github.com/open-telemetry/opentelemetry-service/tree/master/receiver/zipkinreceiver.

I'll close this issue, please subscribe to elastic/apm#98 if you'd like to follow along. No idea when any of the above would become reality, but I think that's the path forward.

[ThomasLohmann]

@axw Hi everyone, I have a general question to the Elastic team: Do you have any plans to integrate the Istio (Envoy) Service Mesh more deeper into the Elastic Stack (Filebeat, Metricbeat, Elastic APM) for observability like Grafana does? Because we observed that the Kubernetes community is moving in the Istio direction for traffic management, observability, security etc. So it would be great if you guys also make it simpler to integrate Istio into your stack. In best case, without Prometheus, Jaeger dependencies, because Metricbeat and Elastic APM provide similar functionality anyway. Please, see a detailed comparison between the Elastic Stack and Grafana Stack for observability in the following link: https://photos.app.goo.gl/TVatniDY9weV3WZE8

Also the OpenTelemetry project could be helpful to move forward: https://opentelemetry.io/

[axw]

@ThomasLohmann nice diagram!

I believe we have all the individual pieces available now:

• Filebeat for logs, with the Envoy module
• Metricbeat for metrics, with the Istio module
• Elastic APM for traces, using the Jaeger integration

Assuming Envoy adds OpenTelemetry instrumentation, then we could also ingest those trace data using the recently implemented OpenTelemetry integration.

No specific plan yet to integrate all of this yet, but I think it would make sense to have some kind of packaging to simplify the configuration. Keep a look out for developments in future releases.

[ThomasLohmann]

@axw I see. But if I use the Istio Module from Metricbeat which needs Prometheus and Elastic APM with the Jaeger integration, I actually need to maintain 4 things (Metricbeat, Prometheus, Elastic APM, Jaeger) which is bad as you can imagine. In best case, you find a way to avoid using Prometheus to store metrics from Istio and instead store them directly in Elastic. The same for traces without the use of Jaeger. That would simplify things dramatically. It could also be a good opportunity for Elastic to put your stack on the Istio integration page like Grafana does. Please, see the following link: https://istio.io/latest/docs/ops/integrations/

[axw]

But if I use the Istio Module from Metricbeat which needs Prometheus and Elastic APM with the Jaeger integration, I actually need to maintain 4 things (Metricbeat, Prometheus, Elastic APM, Jaeger) which is bad as you can imagine.

Metricbeat is able to pull metrics directly from the Prometheus exporters, i.e. from Istio. You could use the Prometheus collector as well (Metricbeat -> Prometheus -> Istio), but that is not necessary, nor what the Metricbeat Istio module does by default.

With the Jaeger integration, Elastic APM Server acts as a stand-in for Jaeger Collector. You still need Jaeger Agent (I think?), but not the full Jaeger architecture.

Anyway, I think it's inevitable that Istio/Envoy will migrate to OpenTelemetry. Once that happens, we can simplify things considerably. We're considering adding support for the OpenTelemetry Protocol (OTLP) in Elastic APM Server, which would then allow us to receive trace data directly from Istio/Envoy. When this all comes to fruition, you would have at most these components:

• Elastic APM
• Metricbeat
• Filebeat

[KarstenSiemer]

Hi! What is the progress of this?

[simitt]

APM Server already supports otlp. It looks like there has also been progress on the Istio/Envoy support for OpenTelemetry (envoyproxy/envoy#9958 (comment)).
@KarstenSiemer are you planning on trying this out?

[KarstenSiemer]

Hello @simitt!
Yes, I am looking into sending traces directly into apm from istio. Currently apm is still interfaced using zipkin from jaeger