[PIP-264] Parent issue for implementation

[asafm]

Overview

This issue will the parent issue tracking the implementation of PIP-264 (PR, issue that was converted to PR): Enhanced OTel-based metric system

Execution Plan

There are two parallel tracks of implementation:

• OpenTelemetry - implementing all changes required to use OpenTelemetry in a low latency system such as Pulsar.
• Pulsar - implementing the design per the parent PIP (264). See the complete design doc here.

Each part is composed of multiple epics. Some require Saga, divided into epics. The contents of each epic - its breakdown into different stories will be inside each epic, with each story having its issues. In Pulsar, many changes will require writing sub-PIPs. They will be linked appropriately inside each story issue.

🔄 = In-progress.

OpenTelemetry track

This track is mainly done in https://github.com/open-telemetry/opentelemetry-java and https://github.com/open-telemetry/opentelemetry-specification.

• 🔄 OTel should be performant enough
  • OTel should be allocation-free on the collection path - asynchronous instruments
    • Learn and draft a plan for what to change
    • OTel Memory allocation test harness (JMH based)
    • Run JMH before changes
    • Modify the code and run JMH on it
    • 🔄 Submit PR and get it approved
      • Memory Mode support: Adding memory mode, and implementing it for Asynchronous Instruments open-telemetry/opentelemetry-java#5709
  • OTel should be allocation free on the collection path - synchronous instruments
    • Exponential Histogram
      • Memory Mode: Adding first part support for synchronous instruments - storage open-telemetry/opentelemetry-java#5998
      • Memory Mode: Adding 2nd part support for synchronous instruments - exponential histogram open-telemetry/opentelemetry-java#6058
      • Memory Mode: Adding 3rd and last part support for synchronous instruments - exponential histogram open-telemetry/opentelemetry-java#6136
    • Explicit Histogram
      • Memory mode: Adding support for synchronous instruments - explicit histogram open-telemetry/opentelemetry-java#6153
    • Counter (Sum aggregator)
      • Memory mode: Adding support for synchronous instruments - Counter open-telemetry/opentelemetry-java#6182
    • 🔄 Last value aggregators
      • Memory mode: Adding support for synchronous instruments - Last Value aggregation open-telemetry/opentelemetry-java#6196
  • OTel memory mode should be supported by all readers and exporters
  • 🔄 OTel should be allocation free on OTLP HTTP Exporter
  • 🔄 OTel should support pushdown predicate
    • OTel specifications should have a pushdown predicate
      • Create issue
        • Proposal: Add filter predicate to MetricReader (push-down predicate) open-telemetry/opentelemetry-specification#3324
      • Garner favorable agreement from the community
      • Write the proposal as PR
        • Add predicate to MetricReader and MetricProducer open-telemetry/opentelemetry-specification#3566
      • Work on PR to get it approved
    • 🔄 Implement spec: MetricFilter open-telemetry/opentelemetry-java#6107
      • Design and approve
      • [] 🔄 Write code with tests
      • Get PR approved
• OTel should be bug-free for the Pulsar use case
  • Exemplars pre-allocates memory even when exemplars are not used open-telemetry/opentelemetry-java#5581
  • Async instruments don't do spatial reaggregation when attributes are dropped open-telemetry/opentelemetry-java#4901
• OTel should allow copying a resource attribute into each outgoing UTS in the Prometheus exporter
  • Add proposal to OTel Specifications and approve it.
    • Allow Prometheus exporter to add resource attributes to metric attributes open-telemetry/opentelemetry-specification#3761
  • Implement specifications in OTel Java SDK
    • Allow Prometheus exporter to add resource attributes to metric attributes open-telemetry/opentelemetry-java#6179
  • 🔄 Improve AutoConfiguredOpenTelemetrySdkBuilder to allow customizing Prometheus exporter before it's being built to allow Pulsar to specify the pulsar.cluster attribute to copy
  • Allow converting ProemetheusHttpServer to its builder so we can set the resource attributes predicate in the auto configured sdk builder.
    • Add toBuilder() at PrometheusHttpServer open-telemetry/opentelemetry-java#6333 (comment)
• OTel should support adding custom authentication to Prometheus exporter
  • Prometheus HTTP endpoint can have custom authentication  open-telemetry/opentelemetry-java#6013

Pulsar track

• 🔄 OTel scaffolding
  • 🔄 OTel should support copying Resource attributes in Prometheus Exporter
    • Create a proposal and approve it: Allow Prometheus exporter to add resource attributes to metric attributes open-telemetry/opentelemetry-specification#3761
    • Implement it in OTel Java SDK: Allow Prometheus exporter to add resource attributes to metric attributes open-telemetry/opentelemetry-java#6108
  • 🔄 OTel layer with matching configuration, which includes Prometheus export and OLTP GRPC export
    • Design PIP
    • Publish and approve PIP
    • 🔄 Implement it - @dragosvictor
  • Decide on naming
    • Decide on the semantic conventions and rules we will apply for the naming. We don’t need a PIP for this, as we’ll introduce the naming changes as part of a PIP
• All Prometheus metrics are in OTel, except BK Metrics API, Plugin metrics, Pulsar Function Metrics
  • Messaging metrics in OTel
• Update All Grafana dashboards.
• Run performance test manually, verifying latency is not impacted up to 10k topics
• Adding BK Metrics API implementation for OTel
• Add support for Pulsar Functions.
• Plugins support for OTel
• Rate should be fully managed.
• Metrics documentation rule (Forcing metrics definition to contain description and units)
• Support metrics in OTel for 100k topics per broker
  • Introduce Topic Metrics Groups
  • Introduce Filtering
  • Performance test to make sure it works just as well as the previous solution
• Support Pulsar authentication on OTel Prometheus exporter
• Deprecate the Prometheus system

[asafm]

In OpenTelemetry track, the support for object pooling to reduce memory allocation by almost 98% was implemented and merged but only for asynchronous instruments. I'm starting to work on doing that for synchronous instruments. In between those PRs I'm using the time to design the first sub PIP: creating the infra needed for any developer to use OTel in Pulsar, including Plugins.

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[asafm]

@tisonkun Can this not be marked stale all the time?

[tisonkun]

@asafm I ever talked about the stale bot, you can resume the discussion and I'm on the side to disable it.

• https://lists.apache.org/thread/tv774jqohdpx8x0dymsskrd90xwwfvgp
• https://lists.apache.org/thread/x2c7xod8y0wvh14nsb6bknf0dq3r9gls
• https://lists.apache.org/thread/0woo9h53t109qsmtxsfqlcxzr16n5mn0

I may not have time to implement something alternatives now, so simply disable it makes sense to me.