Setup prometheus-rules in Mimir

[TheoBrigitte]

Mimir currently runs on gauss and do ingest metrics from other Prometheus servers (#2266 #2265).

In order to replace the current Prometheus setup we need to move the Prometheus Rules evaluation from Prometheus Server to Mimir.

[whites11]

prometheus-operator does not support mimir, we need to check grafana-operator

[QuentinBisson]

The grafana agents supports that out of the box https://gigantic.slack.com/archives/C05AUBA75Q9/p1685709315899309

[TheoBrigitte]

We know grafana-agent supports this and we now have an app for it.
What's the next step here ?

[QuentinBisson]

Review this giantswarm/grafana-agent-app#3, release and deploy it on gauss

[TheoBrigitte]

Also related giantswarm/prometheus-meta-operator#1309

[QuentinBisson]

Prometheus Rules are setup in mimir thanks to the grafana agent in flow mode (currently deployed by hand) using the following command:

helm install grafana-agent grafana/grafana-agent -n monitoring -f configmap.yaml

with configmap.yaml being:

agent:
  configMap:
    content: |
      mimir.rules.kubernetes "local" {
        address = "http://mimir-ruler.mimir.svc:8080/"
      }
controller: deployment # defaults to daemonset which is not useful for our use case

That way, prometheus rules CR are transformed by the grafana agent to prometheus rules sent to the mimir ruler.

In addition, we deployed the mimir alertmanager and a datasource to make sure we can see the alerts.

What's left:

• Deploy the grafana agent in a proper way in Gauss instead of manually. https://github.com/giantswarm/giantswarm-management-clusters/pull/100
• Find how to send the alertmanager config to alertmanager? (mimirtool alertmanager load) done manually on gauss
• Make silence-operator work with mimir? Point silence operator to mimir silence-operator#318
• Alerting rules are missing some labels (external labels actually) https://github.com/giantswarm/sloth-rules/pull/60

Questions:

• Does inhibition rules work in Mimir? Looks supported
• How to load alertmanager config per tenant? We need to use mimirtool

[QuentinBisson]

Rules are missing labels that are added by external labels when using remote write.

All slo alerts should be grouped by them:

(sum(rate(operatorkit_controller_errors_total{container="aws-operator"}[5m])) by (app, cluster_id, installation, cluster_type, region, customer, provider, pipeline))
        /
        (sum(rate(operatorkit_controller_operation_total{container="aws-operator"}[5m])) by (app, cluster_id, installation, cluster_type, region, customer, provider, pipeline))

instead of

(sum(rate(operatorkit_controller_errors_total{container="aws-operator"}[5m])) by (app))
        /
        (sum(rate(operatorkit_controller_operation_total{container="aws-operator"}[5m])) by (app))

[QuentinBisson]

giantswarm/silence-operator#318

[QuentinBisson]

@giantswarm/team-atlas could you take a look at this issue? :)

[QuentinBisson]

Sloth alerts are fixed, the others will be a case 2 case basis:

[QuentinBisson]

Done