Monitoring Flux

[heretic098]

Hi, I hope it's ok to post here, I'm trying to monitor Flux using Google managed prometheus in GKE. The flux docs have a set of instructions for deploying another prometheus stack which I think ultimately comes down to deploying a custom PodMonitor resource. I naively tried changing the apiVersion and kind on the manifest and trying to install it but of course that didn't work :(

Can anyone suggest what the manifest ought to look like based on this hacked version:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: flux-system
  namespace: flux-system
  labels:
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/component: monitoring
spec:
  namespaceSelector:
    matchNames:
      - flux-system
  selector:
    matchExpressions:
      - key: app
        operator: In
        values:
          - helm-controller
          - source-controller
          - kustomize-controller
          - notification-controller
          - image-automation-controller
          - image-reflector-controller
  podMetricsEndpoints:
    - port: http-prom
      relabelings:
        # https://github.com/prometheus-operator/prometheus-operator/issues/4816
        - sourceLabels: [__meta_kubernetes_pod_phase]
          action: keep
          regex: Running

Sorry for the n00b question, I've not worked with prometheus before.

[pintohutch]

Hi @heretic098,

Apologies for the delayed response. So PodMonitoring as a different API spec from the prometheus-operator PodMonitor, which is why you're facing this issue (e.g. there's no namespaceSelector in PodMonitoring).

However, you should be able to modify your resource to look something like:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: flux-system
  namespace: flux-system
  labels:
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/component: monitoring
spec:
  selector:
    matchExpressions:
    - key: app
      operator: In
      values:
      - helm-controller
      - source-controller
      - kustomize-controller
      - notification-controller
      - image-automation-controller
      - image-reflector-controller
  endpoints:
  - port: http-prom
    interval: 30s

I've never used Flux, so I may not have the PodMonitoring totally correct for scraping it, but we've had other folks post about their usage of it, e.g. #364. Maybe they have a PodMonitoring they can suggest.

[heretic098]

Thanks again, I can see the metrics coming through in the project's "monitoring" screen now and and now trying to work out stealing the PromQL from https://github.com/fluxcd/flux2/blob/main/manifests/monitoring/monitoring-config/dashboards/cluster.json to create my dashboard widgets.

[heretic098]

Hi @pintohutch , that does indeed apply and create a PodMonitoring resource in the flux-system namespace. I'll keep an eye on it and see if any metrics appear in GCP managed prometheus. Fingers crossed.

Thank you for your assistance,

[pintohutch]

Glad to hear it!