Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use different cluster job selector for metrics scraping #74

Closed
bsctl opened this issue Aug 21, 2024 · 0 comments · Fixed by #75
Closed

Use different cluster job selector for metrics scraping #74

bsctl opened this issue Aug 21, 2024 · 0 comments · Fixed by #75
Assignees
Milestone

Comments

@bsctl
Copy link
Member

bsctl commented Aug 21, 2024

When enabling Prometheus ServiceMonitor on multiple kamaji-etcd instances, all the ServiceMonitors use the same job selector based on the app.kubernetes.io/name=kamaji-etcd label. This leads Prometheus to consider all the kamaji-etcd instances as part of a single etcd cluster when collecting and displaying metrics.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: roma-capitale
    meta.helm.sh/release-namespace: roma-capitale
  labels:
    app.kubernetes.io/instance: roma-capitale
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kamaji-etcd
    app.kubernetes.io/version: 3.5.6
    helm.sh/chart: kamaji-etcd-0.7.0
  name: roma-capitale-monitor
  namespace: monitoring-system
spec:
  endpoints:
  - interval: 15s
    path: /metrics
    port: metrics
  namespaceSelector:
    matchNames:
    - roma-capitale
  selector:
    matchLabels:
      app.kubernetes.io/instance: roma-capitale
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: kamaji-etcd                             <<<<< this selector
      app.kubernetes.io/version: 3.5.6
      helm.sh/chart: kamaji-etcd-0.7.0

A simple solution is to force Prometheus to use a different label as for example:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: roma-capitale
    meta.helm.sh/release-namespace: roma-capitale
  labels:
    app.kubernetes.io/instance: roma-capitale
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kamaji-etcd
    app.kubernetes.io/version: 3.5.6
    helm.sh/chart: kamaji-etcd-0.7.0
  name: roma-capitale-monitor
  namespace: monitoring-system
spec:
  endpoints:
  - interval: 15s
    path: /metrics
    port: metrics
  jobLabel: app.kubernetes.io/instance    <<<<< this selector
  namespaceSelector:
    matchNames:
    - roma-capitale
  selector:
    matchLabels:
      app.kubernetes.io/instance: roma-capitale
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: kamaji-etcd
      app.kubernetes.io/version: 3.5.6
      helm.sh/chart: kamaji-etcd-0.7.0
@bsctl bsctl self-assigned this Aug 21, 2024
@bsctl bsctl added this to the v0.8.0 milestone Aug 21, 2024
@bsctl bsctl closed this as completed in #75 Aug 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant