Define the list of components (receivers, exporters, processors, extensions) to support in core

[bogdandrutu]

Based on the decision made in #2542 we will not have multiple go-modules in core, and we should be very careful with the binary size as well as the dependencies that we bring to core.

Components that are now in core and we decide to remove them will be moved in contrib (where we have to have a "support level" defined, and these components will be the highest level "a.k.a maintained by the core maintainers and approvers". The support level and all other details will be decided later when we start the work on GAing the contrib repo.

@tigrannajaryan and I discussed about this offline, and we would like to get feedback about this proposal. Here is the proposed list of components (all helpers will also be in core):

• Receivers
  • OTLP
  • File with OTLP json/proto - not available now but we will add it.
• Processors
  • Memory limiter
  • Batch
  • Resource transformation (name to be decided)
  • Simple Traces transformation (name to be decided, exact list of features to be decided)
  • Simple Metrics transformation (name to be decided, exact list of features to be decided)
  • Simple Logs transformation (name to be decided, exact list of features to be decided)
• Exporters
  • OTLP grpc and http
  • File with OTLP json/proto - needs to add support for proto, file rotation, etc.
  • Debug
• Extension
  • Z-Pages
  • Health
  • PProf (may be consider to merge with z-pages, but we can discuss later)

[jpkrohling]

I miss the receivers and exporters for interop with other open-source protocols and formats, such as Jaeger and Zipkin. Leaving my Jaeger-maintainer hat aside for the moment, I believe that we should aim to receive data in formats that are currently the standard, like the aforementioned. Without them, people will be "forced" to use a downstream distribution in an environment where distributed tracing already exists in a form or another.

About the exporters, we can argue that there are enough vendors out there with support for ingesting OTLP so that leaving Jaeger and Zipkin out might be acceptable. However, I would still wish we had Jaeger and Zipkin exporters, as there are vastly more offerings able to ingest those two formats.

I cannot speak for the Zipkin folks, but we have enough Jaeger maintainers interested in keeping Jaeger receivers/exporters functional in OpenTelemetry so that it doesn't become a burden on the otelcol maintainers.

cc @albertteoh, @joe-elliott

[joe-elliott]

I am in agreement that including other open source protocols available in the core collector has value. I'm assuming that it has some, but minimal impact on the final binary size?

One of the reasons I regularly point people to use the collector as a tool is b/c of its advanced interoperability with all common tracing OS protocols out of the box. This will still technically be possible but will require a custom build?

[bogdandrutu]

@joe-elliott @jpkrohling we need to ensure that we make a difference between build (binaries we produce and put in different registries) and modules/packages.

We will definitely have official builds for:

1. light otel collector (only OTLP and File support)
2. Standard otel collector (support for all open-source standards jaeger, zipink, prometheus)
3. All contrib packages

The biggest problem on the binary size is Prometheus, it is about ~80MB if I remember correctly :(

[bogdandrutu]

@jpkrohling as mentioned in the issue the Jaeger will be tier 1 support in OpenTelemetry and was not a burden in general, but I want to limit the code that we ship as "core". We can discuss if we want to keep Jaeger directly in this repo or as a tier 1 supported thing in contrib, or even a different repository for tier1 supported components.

[tigrannajaryan]

Really Prometheus size is huge. We are just trying to come up with reasonable rules without arbitrarily choosing components and one idea Bogdan and I had was that only OTLP as the truly native format lives in the core and the rest go to contrib. Not a strong opinion, I am also happy to consider keeping some more components in the core, especially given that we have willing maintainers for those components.

I would prefer not to have 3 different builds for now and stay with 2 if possible. 3 builds is just more work for everyone. I think that should be the last resort.

[jpkrohling]

we need to ensure that we make a difference between build (binaries we produce and put in different registries) and modules/packages.

Indeed, I mixed up both.

The biggest problem on the binary size is Prometheus, it is about ~80MB if I remember correctly :(

Prometheus is part of the metrics part, which isn't a concern for GA, as I understand. Perhaps impact on the binary size could be a criterion for inclusion in the main distribution?

only OTLP as the truly native format lives in the core and the rest go to contrib

Would you have anything against if we hosted the Jaeger components in a repository under the jaegertracing organization instead of contrib?

[pkositsyn]

Would you have anything against if we hosted the Jaeger components in a repository under the jaegertracing organization instead of contrib?

I think it is a big decision to let or not to let contrib components be maintained in other places. Maybe create a separate issue for discussion?

[alolita]

@bogdandrutu based on our triage discussion, can you move this issue to the phase 2 list. Thanks!

[alolita]

@bogdandrutu should we create a separate issue for discussion as recommended above?

[alolita]

Superceded by issue #3474 to achieve a stable Collector core for upcoming trace stable release.