Integration with existing trace libraries

[adam-mccormick]

Projects like opentelmetry provide prebuilt instrumentation adapters for a whole range of libraries like MySQL, express, cloud SDKs, socket.io etc. It would convenient to use these tools in Moleculer without having to abandon the built-in tracing API provided by the framework.

Perhaps there's a way to share the current trace and span context between Moleculer and another library like OpenTelemetry and allow both to update the trace context.

A better idea might be to have a configurable SpanFactory that can create span instances that follow the existing Moleculer API but can use delegate span objects from other frameworks.

[icebob]

The Moleculer built-in tracing feature is a complete solution like the OpenTelemetry lib. Yeah, Moleculer Tracer doesn't have instrumentation for external libs and I don't plan to add it because it's a big "hacking" job, a lot of time, and doesn't give extra to the framework itself.

Instead, my suggestion is to create a Moleculer instrumentation for OpenTelemetry. Similar to what dd-trace does.

It can be a middleware (based on the built-in tracing middleware) which wraps the localAction and localEvent hooks and create OT spans instead of Moleculer Tracer spans.

[intech]

I agree that this is a lot of work.

I have an idea of how we can solve this by creating a bridge for connecting external shimmer/wrap tracing modules, according to the principle, as it is now done in the Elastic APM node.js agent.
The idea is to forward moleculer tracing methods and they can interact with these modules through a normal connection.

I’m also waiting and hoping for the node.js API that in the end, everything will come to a single channel for application tracing, and we won’t have to invent our own solution in each library.

[icebob]

I made an OpenTelemetry middleware and a demo Moleculer project which demonstrates my suggestion. It uses B3 propagator which pushes tracing info into ctx.meta, so it can connect the traces of remote actions as well. And also can connect the trace spans between built-in instruments and Moleculer action calls.

https://github.com/icebob/moleculer-opentelemetry-demo

The main things are in:

• https://github.com/icebob/moleculer-opentelemetry-demo/blob/master/opentelemetry.middleware.js
• https://github.com/icebob/moleculer-opentelemetry-demo/blob/master/moleculer.config.js

My experiences with OT:

1. OpenTelemetry documentation is very simple, not easy to make complex solutions. I had to check the source code many times to find out how it works and it took ~6 hours from my time to make a working demo project :(
2. Jaeger accepts the service name only once (in getSender which generates only one per process), so this is why all spans' service is moleculer. Maybe Zipkin exporter is better.

[vkruoso]

Hi @icebob @intech, thanks a lot for the work. Was looking around your demo and trying to figure out a way to get the span service name to be the correct one. Trying with Zipkin exporter and Jaeger, gives the result below, which looks better. It is just missing the correct name for the specific OT libraries spans. Maybe is related to this: open-telemetry/opentelemetry-js#2606. If you have any pointers, would be appreciated. If I manage to get it working will let you know!

[icebob]

Yeah, this is the reason. Jaeger exporter creates a Sender which contains the service name and this sender cached and not created again. So all spans from the given process will use the same service name. This issue is solved in the built-in Moleculer Jaeger exporter here:

moleculer/src/tracing/exporters/jaeger.js

Lines 162 to 177 in 1fa2440

	getTracer(serviceName) {
	if (this.tracers[serviceName]) return this.tracers[serviceName];
	const sampler = this.getSampler(serviceName);
	const reporter = this.getReporter();
	const tracer = new Jaeger.Tracer(
	serviceName,
	reporter,
	sampler,
	Object.assign({ logger: this.logger }, this.opts.tracerOptions)
	);
	this.tracers[serviceName] = tracer;
	return tracer;
	}

[vkruoso]

Sorry for the delay here. Thanks a lot @icebob and others here. I did what you suggested and it worked for us! We are running only one service per process at this moment, so it simplifies things a little, but tracing is working nicely with NewRelic, and locally with Jaeger as well using the Zipkin format. As a side note, just had to do small changes to your code in the error handling part (to close the span to correctly report the error) and in some parameters of the traces as NewRelic was not linking some of them.

[daniel-white]

Hey @icebob would you consider making this part of OT's contrib repo? That way it can grow with the other instrumentation packages

[Aaron-Ritter]

Nice example! btw. you could use the standard OTLP exporter and run e.g. https://raw.githubusercontent.com/grafana/docker-otel-lgtm/refs/heads/main/run-lgtm.sh locally. Same topic with 1 process 1 service but one less component in the mix.

by the way one of the reasons why you see a single span rather than multiple ones is described well here https://stackoverflow.com/questions/72867313/opentelemetry-open-span-in-one-process-and-close-it-in-another. and this is partially just because how moleculer works. And at some point profiling will allow more detailed insight into the different spans https://opentelemetry.io/blog/2024/profiling/