Spike for pipeline healthyness as part of the status

[a-thaler]

Description
For realizing #425 a concrete concept is agreed on. In order to determine if the solution will fit to the problem well and finding obstacles, a spike should be done which brings all building blocks together and allows a verification of the e2e flow.

Run an e2e spike focussing on the business functionality, collect metrics and leverage them to serve the new API:

• Have a prometheus running (no automation needed yet)
• Configure prometheus so that well-defined alerts are available for the error situations
  • every alert should be one error situation, per pipeline if applicable
• Implement the alert webhook to trigger reconcilation, triggering only needed reconcilations (do not trigger traces if logs are firing)
• Define a concrete status API (for the new Status AP of metrics)
• Have a look in to the old status API and get a feeling on integrating it as well
• Implement the reconcile logic and set the status (for the new Status AP of metrics)
• Show the status in the dashboard
• Reduce the amount of timeseries as much as possible

Reasons

Attachments

Release Notes

[rakesh-garimella]

ADR for not making prometheus part of service mesh: #758

[rakesh-garimella]

required istio settings

Prometheus Wont be part of the istio service mesh. So no setting required.

what if the status is in an alert and prometheus restarts, will a recovery from the alert be detected

In case of prometheus restart we see error failed to query Prometheus alerts: Get \"http://prometheus-server.default:80/api/v1/alerts\" and when the prometheus is healthy again we see the alerts still shown.

can there be flaky alerts

Could not reproduce this as when a alert is triggered as per current query it takes atleast 5 mins for the alert to be resolved. So if we keep the interval on which the alert is measured long enough then we should not see falkiness

what is the actual footprint of prometheus?

[rakesh-garimella]

The ADR for health api: #802 has been merged so closing the issue