PoC: Assess the impact of direct Prometheus queries on reconciliation duration

[skhalash]

Description

This effort is a follow-up to the work done in #516 we need to understand the impact of direct Prometheus queries on reconciliation duration.

Acceptance Criteria

• Build a PoC that setup a simple Prometheus alongside the Telemetry Manager
• Develop a set of PromQL queries based on OTel Collector metrics to detect specific scenarios, such as backend connectivity issues, backpressure/throttling due to backend overload, and reaching the ingestion limit of a pipeline.
• Check the different patterns we identified in the ADR for impact on reconcilation, especially the duration for one reconcilation
• Document the decisions in the ADR as part of the story

Related Issues
#425

[skhalash]

Selected Queries:

rate(otelcol_exporter_send_failed_metric_points[5m]) > 0
rate(otelcol_exporter_enqueue_failed_metric_points[5m]) > 0
(otelcol_exporter_queue_size/otelcol_exporter_queue_capacity)*100 > 90

rate(otelcol_processor_dropped_metric_points[5m]) > 0
rate(otelcol_processor_refused_metric_points[5m]) > 0

rate(otelcol_receiver_refused_metric_points[5m]) > 0

Prometheus Query Execution Time

Testing individual Prometheus queries has demonstrated that the execution time on k3d is negligible, typically in the range of 10s of milliseconds. In a real cluster environment, the execution time might be somewhat higher, b ut still primarily influenced by network latency.

[skhalash]

To wrap it up, the consensus was to implement the integration of Prometheus and Telemetry Manager using the Alerting approach.

• ADR
• PoC