Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ServiceMonitor misconfiguration #357

Open
VLZZZ opened this issue Jan 17, 2024 · 1 comment · May be fixed by #363
Open

ServiceMonitor misconfiguration #357

VLZZZ opened this issue Jan 17, 2024 · 1 comment · May be fixed by #363
Assignees

Comments

@VLZZZ
Copy link

VLZZZ commented Jan 17, 2024

Hi!
I've just found that litmus ServiceMonitor for metrics collection seems to be misconfigured
I deep dived a bit and it looks like ServiceMonitor selector:

spec:
  endpoints:
  - path: /metrics
    port: http
  namespaceSelector:
    matchNames:
    - litmus
  selector:
    matchLabels:
      app: litmus
      app.kubernetes.io/instance: litmus-chaos
      app.kubernetes.io/name: litmus

which is resulted in litmus-monitor service (kubernetes service entity).
That has a Pod selector:

  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: http
  selector:
    app: litmus
    app.kubernetes.io/instance: litmus-chaos
    app.kubernetes.io/name: litmus

But this selector covers both litmus and litmus-monitor pods.

But only litmus-monitor-* has port named http at 8080 and serves prometheus metrics
While litmus-* pod has http at 80 with no prometheus metrics. (litmus pods can server default go metrics at 8080 but it's a different story and I don't think that we need this)

    name: chaos-operator
    ports:
    - containerPort: 80
      name: http
      protocol: TCP

I believe we need to narrow selector to match litmus-monitor only.

Or to allow additionalLables for the selector

  1. https://github.com/litmuschaos/litmus-helm/blob/31a0a30a370c64de6227bfb0ff10035c4b[…]dcd8c/charts/litmus-core/templates/exporter-servicemonitor.yaml
  2. {{- define "litmus.selectors" }}
@Calvinaud Calvinaud linked a pull request Feb 28, 2024 that will close this issue
3 tasks
@Calvinaud
Copy link
Contributor

Calvinaud commented Feb 28, 2024

Hello,

I create a PR for this.
In the litmus-agent, the serviceMonitor and Service only select the exporter pods. Make sense the litmus-core should do the same.
If we need to retrieve the metrics of the operator we probably need to add a podMonitor instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants