[alertmanager] Add prometheus servicemonitor support

[Arabus]

What this PR does / why we need it

Add servicemonitor support to alertmanager

Which issue this PR fixes

• fixes [alertmanager] Provide servicemonitor template #4630

Special notes for your reviewer

Checklist

• DCO signed
• Chart Version bumped
• Title of the PR starts with chart name (e.g. [prometheus-couchdb-exporter])

[Arabus]

@monotek it appears the lint check failed for no apparent reason?

[Arabus]

nvm i see the errors now; though they are not related to anything I touched and I don't see what the linter wants me to do. there appear to be some problems wit the comments around l 115 and 116 🤷

[Arabus]

mucking around with yamllint did the trick but now I face the consequence of adding a resource that requires the prometheus crd for a service monitor... so either I add the crd chart s as a dependency (preferred) or exclude the resource based on a capabilities test.

WDYT?

[Arabus]

well should work now - please have a look

[DrFaust92]

my 2 cents,
I think the AM chart shouldnt bundle crds, same as prom chart (non operator) doesnt assume CRDs or tries to install them. it should "assume" their existence and install svc monitor if api is available, like so

helm-charts/charts/prometheus-fastly-exporter/templates/servicemonitor.yaml

Line 2 in 41a8b07

{{- if and ( .Capabilities.APIVersions.Has "monitoring.coreos.com/v1" ) ( .Values.serviceMonitor.enabled ) }}

this way its loosely coupled

[monotek]

see code comments...

[Arabus]

my 2 cents, I think the AM chart shouldnt bundle crds, same as prom chart (non operator) doesnt assume CRDs or tries to install them. it should "assume" their existence and install svc monitor if api is available, like so

helm-charts/charts/prometheus-fastly-exporter/templates/servicemonitor.yaml

Line 2 in 41a8b07

{{- if and ( .Capabilities.APIVersions.Has "monitoring.coreos.com/v1" ) ( .Values.serviceMonitor.enabled ) }}

this way its loosely coupled

That was the other alternative I considered; I'm fine with either way. This contains the 'risk' of not deploying the servicemonitor even though someone enabled it though. I have added an additional if - fail instead; that way one has to have the capability or explicitly fake it with --api-versions

[Arabus]

I noticed a side effect of the current change; due to the chosen label selectors we get duplicate metrics. The headless and regular services have the same labels, which is why the servicemonitors label selector matches both.

In order to distinguish them, I could add a label to the headless service. I could also switch to using a pod monitor.

There is also the option of using a targetLimit though I have never used that and will need to test if it does what we want.