Adding TraceExporter from instrumented code

[tobias-]

In code recently written, we want to push the traces to GCE stackdriver. As I started assuming that I could do without any instrumentation (nor even knew it existed), it was written in our codebase. Unfortunately, this became a dead end, because of over reliance on local coroutine scopes and opentelemetry-java-instrumentation was found.

So there are two problem I want to solve by adding a trace exports in the instrumented code:

1. Use the code I wrote
2. Able to select which exporter to use ("logger" vs "stackdriver") depending on environment, ie local or cloud without duplicating that logic.

The code I wrote unfortunately depends on a lot of stuff not easily extracted into a separate jar ready for -Dotel.exporter.jar consumption (since it depends on Kodein injections, config read in special way, other projects and loads of dependencies)
The question is, can I do this in some way, or am I forced to try to extract the client into a separate project with config logic, environment select, and kodein code copied or rewritten?

[anuraaga]

Hi @tobias- - can you first clarify, it sounds like you wrote a custom exporter for Stackdriver, meaning you aren't using the official exporter here?

https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/tree/master/exporters/auto

And to confirm, you indeed definitely do need to use the code you wrote instead of switching to this? Ideally no, since I'm not sure of a better answer here. otel.exporter.jar is our mechanism for users running a custom exporter. We have another mechanism using custom agent distros which you may want to look at, but it's far more complicated so may be a net loss.

https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/master/examples/distro

Let me know if you have any initial thoughts or we can try to come up with something.

[tobias-]

can you first clarify, it sounds like you wrote a custom exporter for Stackdriver, meaning you aren't using the official exporter here?

https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/tree/master/exporters/auto

Correct. As I said, we started out without even knowing about the agent, never mind the extension to it. I did find the stackdriver driver, but after looking at it's pretty impressive amount of Google dependencies that would cause havoc, we quickly decided against using it and just copy the code for translating the objects.

The auto project is something I didn't know about until you told me just now. That will help a lot. I only read stuff like Not currently ready as a package to be imported in trace/ under installation and figured I'd need to roll it myself anyways.

And to confirm, you indeed definitely do need to use the code you wrote instead of switching to this? Ideally no, since I'm not sure of a better answer here. otel.exporter.jar is our mechanism for users running a custom exporter. We have another mechanism using custom agent distros which you may want to look at, but it's far more complicated so may be a net loss.

No, "need" I do not, but it would definitely be preferable. Having e.g. auth set up in another way isn't ideal, but would be doable.The problem of selecting environment would have to be moved to a startup shell script (not ideal either). After thinking and looking through the auto code, it definitely seems that this is the only reasonable way forward though.

https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/master/examples/distro

Let me know if you have any initial thoughts or we can try to come up with something.

Thanks, much appreciate it. I actually looked at the "distro" yesterday after reading the #1945 post and it will be my backup solution, but I hope (and believe) auth will work out.

[anuraaga]

Thanks this sounds like a general concept of wanting more control of the agent configuration from apps. I think this is a reasonable request that many users will want. In the meantime it's good to hear that, even though annoying leaking into startup scripts, you do have some options for using the official stackdriver exporter. I don't want to have to have users rewrite everything just to plug in configuration and hope to be able to deprecate any needed shell script hacks :)

Filed #2002 to see what is possible

[tobias-]

I still see the point of having more control of the code. I'll write my suggestions/use cases into #2002 as I come across them.

As far as I can tell now, with relatively small changes I think (with my extremely limited agent knowledge) that by just adding the ability to set noop exporters.

I actually managed to get it into the code pretty cleanly, and it actually works (!?) with GoogleCloudPlatform/opentelemetry-operations-java#63 . Now I just need to figure out how to do what everybody else wants, e.g. filter spans for health checks. First version will probably be a hack in https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/tree/master/exporters/auto since we need to compile that anyway (it's just 0.11.0 compatible in the master branch).