nrpe checks fail after adding relabel_configs in prometheus

[ahmadfsbd1]

Bug Description

After adding relabel_configs in cos-prometheus through prometheus-scrape-config-k8s charm, the nrpe checks fail. When the relabel_config is removed, the nrpe checks and alerts will work as expected.

To Reproduce

1. Deploy the standard cos
2. Deploy nrpe charm in a separate model with relation to cos-proxy forwarding to cos-prometheus
3. Add the relabel_config to the cos-prometheus using the prometheus-scrape-config-k8s-charm.
   relabel_configs: |-
   - target_label: 'alerting_system'
   replacement: 'nfvi-cos'
   - target_label: 'site_identifier'
   replacement: 'site1'
4. Observe that after the prometheus has resstarted and settled, the nrpe checks are failing.

Environment

juju 3.4/stable, MAAS 3.4/stable

alertmanager alertmanager-k8s latest/edge 128
catalogue catalogue-k8s latest/stable 43
grafana grafana-k8s latest/stable
loki loki-k8s latest/stable
prometheus prometheus-k8s latest/stable
scrape-interval-config-monitors prometheus-scrape-config-k8s latest/edge
traefik traefik-k8s latest/stable 194

cos-proxy-monitors cos-proxy latest/edge 101
nrpe nrpe latest/stable

Relevant log output

See the images in 'To Reproduce'

Additional context

No response

[simskij]

cc @sed-i

[sed-i]

@simskij was this the issue you fixed by using metrics_relabel_config instead of relable_configs?