Ingesting metrics into Mimir

[TheoBrigitte]

Configure our current Prometheus servers to send metrics to Mimir.

This can be achieved via our RemoteWrite CR, see https://docs.giantswarm.io/getting-started/observability/monitoring/prometheus/remotewrite/

[hervenicol]

I created this mimir-remotewrite.yaml:

apiVersion: monitoring.giantswarm.io/v1alpha1
kind: RemoteWrite
metadata:
  labels:
    application.giantswarm.io/team: atlas
  name: mimir
  namespace: mimir
spec:
  clusterSelector: {}
  remoteWrite:
    name: mimir
    queueConfig:
      capacity: 10000
      maxSamplesPerSend: 1000
      minShards: 10
    tlsConfig:
      insecureSkipVerify: false
    url: http://mimir-gateway.mimir.svc/api/v1/push

It sends all metrics from all prometheis to the local Mimir.

It's currently rate-limited by Mimir, but that's a matter of Mimir tuning now.

[QuantumEnigmaa]

Should we add the remotewrite in the chart templates ?

[hervenicol]

Should we add the remotewrite in the chart templates ?

I don't think so, because it's quite specific to how we use it. Customers don't really care about that.
Just like the current remotewrite to grafana cloud, it should be managed separately.
Maybe in the prometheus-remotewrite app, like the grafana-cloud one.

[QuantumEnigmaa]

Sure. Makes sense

[QuantumEnigmaa]

So whar's left to be done here ?

[hervenicol]

I think that's good for now as it's documented in the mimir-app repo.