Auto-instrumentation as a Collector receiver

[damemi]

We could in theory expose a few parts of our internal code to make an auto-instrumentation receiver in the collector.

See #1025 for the code changes to our repo and the WIP collector changes in open-telemetry/opentelemetry-collector-contrib@main...damemi:opentelemetry-collector-contrib:goautoreceiver

This would make it so you don't need to run an extra agent alongside a collector. It also makes it a lot easier to configure auto-instrumented telemetry with collector processors.

The main change is allowing a custom OpenTelemetry Controller (to export traces as ptrace objects to the next consumer, rather than directly to an endpoint). We might not even need this if you can just point the default Controller to the collector's OTLP endpoint (ie, back to itself). But I think building it into the pipeline is the better approach.

What do people think about this idea?

[RonFed]

Is the use case for self tracing of the collector process?
If so, I think there will be some kind of positive feedback loop of exporting traces --> traces are automatically created for the exporter operation --> these traces are exported and this loop can go forever.
But maybe I did not understand the use case.

[damemi]

The idea is to instrument user apps. Right now, the general approach with auto-instrumentation is to run your app+instrumentation as a sidecar. If you're running the collector as another sidecar, that adds complexity (and you have to coordinate startup/shutdown timing).

The Collector could just embed the auto-instrumentation libraries as a "receiver". Then the collector is an auto-instrumentation agent itself. I think this would only really work for Go, since that's what the collector is written in.

[rogercoll]

+1 on this idea. Firstly, because it won't affect this repository (except the new exposed interfaces), but it will allow other applications to be built on top (as the collector receiver).

I think that having a receiver would ease the usage of this auto instrumentation as it will remove the side agent complexity in VM environments. In Kubernetes, I think the Operator approach would still be the recommended as automatically injects the sidecar agent to the user's app Pod (required because it needs access to the user's app binary). Using the collector's receiver in K8s might be a bit more challenging and make the user apps deployment dependent to the collector.

Does the package require CGO? https://github.com/open-telemetry/opentelemetry-collector/blob/v0.109.0/CONTRIBUTING.md#using-cgo