Advanced pipeline status based on data flow

[a-thaler]

Description
The telemetry pipelines are deploying active components to a cluster which are dispatching data to 3party services. Any kind of interruptions can happen which prevents a successful delivery of the data. Retries and buffering will introduce short-term resilience. Still, there will be situations where data cannot be delivered and the user needs to get notified about that to be able to react.

The typical situations causing problems are:

• Connectivity issues to the 3party service
• Backpressure/throttling caused by overload of the 3party service
• Ingestion limit of a pipeline is reached (even with auto-scaling, there will be a maxReplica setting)

These situations can be observed by collecting metrics of the relevant components, a documentation for that was recently added.
However, that approach is cumbersome as the user needs to know what services to scrape and need to filter the relevant metrics. Also, these details are more internals which might change in future.

Goal
The telemetry-manager is managing the lifecycle of pipelines and should be the only place knowing how to interpret the metrics of the components. Any problematic situation should be reported in the pipeline and module status as warning so that a user can easily detect the problem. Also, custom metrics should be exposed via a dedicated endpoint (only returning relevant user facing metrics) which will be maintained long-term, even if internals will change. Another typical channel for notification about problems could be to emit a k8s event in case of turning a pipeline into an unhealthy mode.

Criterias

• As a user I can detect the 3 described situations with my pipelines as part of the pipeline and module status. The description is good enough to identify the problem. The status is documented well.
• As a user I can collect metrics around the new status to observe the 3 situations and being able to setup alerts in my monitoring backend. (to be covered by Telemetry module status as metric input to enable dashboarding and alerting on it #728)

Ideas
The operator could scrape all active components, interpret the data, to report status and custom metrics. For that, also a prometheus sidecar could be used which does the scrape job in a generic way, so that the operator can just use PromQL at any time to do a query. Hereby, a question is if historical data can be beneficial.

Potential metrics to expose

Metric	Meaning	Goal
telemetry_tracegateway_input_accepted	counter indicating successful ingestion rate	observe if data is arriving
telemetry_tracegateway_input_refused	counter indicating a refusal on ingestion side	observe if the component is overload and requires scaling
telemetry_tracegateway_output_dropped (pipeline=xxx)	counter indicating a drop of data on exporter side	observe if the backend has backpressure
telemetry_tracegateway_output_failed (pipeline=xxx)	counter indicating a unrecoverable fail of data on exporter side	observe if there are data consistency problems (400 responses)
telemetry_tracegateway_output_send (pipeline=xxx)	counter indicating the export rate	observe if data can be exported

Potential status API
New reasons for the module conditions should be introduced for the new situation a pipeline can be in. For the TraceComponentsHealthy condition new reasons could be:

Condition status	Condition reason	Message
False	TraceGatewayIngestionThrottling	The gateway cannot handle the incoming load. Please scale up manually
False	PipelineDropsData	The configured backend of pipeline XX drops data. Please check if the configured backend can handle the load.

Items

• Preparation:
  • Investigate the area on possible solutions (docs: ADR: Trace/Metric Pipeline status based on Otel Collector metrics #516)
  • Make required PoCs for a concrete proposal (PoC: Assess the impact of direct Prometheus queries on reconciliation duration #526)
    - What situations we want to detect and is it possible using this approach (what metrics and labels we need to collect)
    - have a simple base setup deploying a Prometheus and start collecting the needed metrics
    - add metric collection to identify the typical notification situations and filter relevant labels
    - add alert rules for that situations with important labels
    - what is the query performance? Is it good enough to call it as part of reconciliation?
    - Can we base the queries on alerts only?
    - Will recording rules help us in contrast do do raw queries?
  • Make an architectural proposal
    • Introduce Prometheus as a short-term metric storage (non-persistent) or do the scraping custom
    • How to integrate into the control loop for the resources, the pipeline controllers should not do active calls to some endpoints as it will delay the reconciliation. Can we have a new controller doing the scraping only and storing the results into a dummy CRD?
  • Validate the proposal with a spike implementation (Spike for pipeline healthyness as part of the status #713)
    • What footprint we need to expect (amount of timeseries/memory/cpu)
    • Come up with a concrete API proposal
    • Validate the usability (how does it feel in the dashboard)
    • Think of corner situations like flapping alerts
  • Come up with a hardened Prometheus setup (Spike for hardened prometheus setup #770)
  • Understand and define metrics and alerts for Fluent Bit, to support advanced status on the LogPipeline as well (Metrics and alerts for Fluent Bit in advanced status #904)
• Implementation
  • Introduce a new feature flag and manage (create/update/delete) a Prometheus doing nothing (Deploy the self-monitor on pipeline activations #807)
  • Report self-monitor alerts to the pipeline status (Report self-monitor alerts to the pipeline status #822)
  • Custom image + experimental channel (Advanced status feature on development release with custom image for self-monitor #871)
  • Add webhook for the fast trigger of reconciliations (Add webhook to trigger status reconciliations faster #903)
  • Support for FluentBit (Support for fluentbit in advanced status handling #917)
  • Have dashboard additions, documentation, messages (Document advanced status API #918)
  • Integration tests (e2e) (Full test coverage for advanced status #919)
  • Monitoring of self-monitor (Self-Monitor monitoring #920)
  • Performance test and optimization (Performance test and optimize self-monitor #1002)
• Rollout (Advanced pipeline status based on data flow as part of regular release #1093)
• Follow-ups
  • Support for K8s events (Understand and design kubernetes events for pipeline status #787)
  • Add verification steps to the integration guides which will ensure that the pipelines are healthy from the data flow perspective
  • Have the module status observable via a MetricPipeline (Telemetry module status as metric input to enable dashboarding and alerting on it #728)

[a-thaler]

As part of the investigations we decided that the main interface should be the module status indicating the error situations. With that the status is accessible via UI and CLI automatically.

On top, we need a generic way to enable the user to collect the status of each module listed in the Kyma CR in a consistent way, most probably without exposing custom metrics via the telemetry-manager. The story of collecting the status will be covered in a dedicated epic: #728

[a-thaler]

We planned to finish the open issues within next 2 weeks, so that a promotion to the regular release can be started afterwards

[a-thaler]

Feature will be part of tomorrows release