Keep searching for job/instance if multiple job/instance matchers present

[v-zhuravlev]

This improvement would keep searching for job/instance matchers in PromQL query instead of failing on the first match.

In this case [target-job-rule] check would not fail if multiple job/instance matchers are present and expected value (i.e job=~"$job") is within the second, not the first match.

This should fix it when job is present in both groupLabels and filteringSelector.

For example:

$instance"}': job selector is .*windows.*, not $job
[target-job-rule] 'Windows overview': Dashboard 'Windows overview', panel 'Disk C: size', target idx '0' invalid PromQL query 'windows_logical_disk_size_bytes{volume="C:", job=~".*windows.*",job=~"$job",instance=~"$instance"}': job selector is .*windows.*, not $job
[target-job-rule] 'Windows overview': Dashboard 'Windows overview', panel 'CPU usage', target idx '0' invalid PromQL query '100 - (avg without (mode,core) (rate(windows_cpu_time_total{mode="idle", job=~".*windows.*",job=~"$job",instance=~"$instance"}[$__rate_interval])*100))': job selector is .*windows.*, not $job
[target-job-rule] 'Windows overview': Dashboard 'Windows overview', panel 'CPU usage', target idx '0' invalid PromQL query '100 - (avg without (mode,core) (rate(windows_cpu_time_total{mode="idle", job=~".*windows.*",job=~"$job",instance=~"$instance"}[$__rate_interval])*100))': job selector is .*windows.*, not $job
[target-job-rule] 'Windows overview': Dashboard 'Windows overview', panel 'Memory usage', target idx '0' invalid PromQL query '100 - windows_os_physical_memory_free_bytes{job=~".*windows.*",job=~"$job",instance=~"$instance"} / windows_cs_physical_memory_bytes{job=~".*windows.*",job=~"$job",instance=~"$instance"} * 100': job selector is .*windows.*, not $job
[target-job-rule] 'Windows overview': Dashboard 'Windows overview', panel 'Memory usage', target idx '0' invalid PromQL query '100 - windows_os_physical_memory_free_bytes{job=~".*windows.*",job=~"$job",instance=~"$instance"} / windows_cs_physical_memory_bytes{job=~".*windows.*",job=~"$job",instance=~"$instance"} * 100': job selector is .*windows.*, not $job
[target-job-rule] 'Windows overview': Dashboard 'Windows overview', panel 'Memory usage', target idx '0' invalid PromQL query 'windows_cs_physical_memory_bytes{job=~".*windows.*",job=~"$job",instance=~"$instance"} - windows_os_physical_memory_free_bytes{job=~".*windows.*",job=~"$job",instance=~"$instance"}': job selector is .*windows.*, not $job
[target-job-rule] 'Windows overview': Dashboard 'Windows overview', panel 'Memory usage', target idx '0' invalid PromQL query 'windows_cs_physical_memory_bytes{job=~".*windows.*",job=~"$job",instance=~"$instance"} - windows_os_physical_memory_free_bytes{job=~".*windows.*",job=~"$job",instance=~"$instance"}': job selector is .*windows.*, not $job
[target-job-rule] 'Windows overview': Dashboard 'Windows overview', panel 'Memory usage', target idx '1' invalid PromQL query 'windows_cs_physical_memory_bytes{job=~".*windows.*",job=~"$job",instance=~"$instance"}': job selector is .*windows.*, not $job

[Dasomeone]

This LGTM, will test the other PR in the morning :)

[v-zhuravlev]

@rgeyer can we merge this?

[rgeyer]

@rgeyer can we merge this?

@Dasomeone and/or @v-zhuravlev, can I get a review of #208 before I merge this? It is a superset of #181 that's been in the works for a long time, so an in-depth review isn't exactly necessary (tho appreciated). It is a bit of a monster and we need to get it merged before much more changes in the repo.

LMK if you have any questions! :)

[rgeyer]

Actually, ignore me, this one is super straightforward and won't cause any conflicts. I'm just hyper sensitive to changes against main right now until we get that other PR settled.