ROX-15234: Prometheus metric for pause-reconcile instances

[vladbologa]

Description

This PR adds a Prometheus metric that exposes the pause-reconcile instances managed by fleetshard-sync.

The metric is implemented as a Gauge, where value 1 means an instance is pause-reconcile'd, and 0 means it's not. Values are labeled by the instance name.

See also stackrox/rhacs-observability-resources#102

Checklist (Definition of Done)

• Unit and integration tests added
• Added test description under Test manual
• Documentation added if necessary (i.e. changes to dev setup, test execution, ...)
• CI and all relevant tests are passing
• Add the ticket number to the PR title if available, i.e. ROX-12345: ...
• Discussed security and business related topics privately. Will move any security and business related topics that arise to private communication channel.
• Add secret to app-interface Vault or Secrets Manager if necessary
• RDS changes were e2e tested manually
• Check AWS limits are reasonable for changes provisioning new resources

Test manual

Manual testing:

make deploy/dev
kubectl -n acsms port-forward fleetshard-sync-5479d64d89-v4mph 8080:8080

Then opened http://localhost:8080/metrics in a browser and checked that the new metrics are there and have correct values, before and after adding the pause-reconcile annotation, e.g.:

./scripts/create-central.sh
# edit the central and add annotation: "stackrox.io/pause-reconcile: "true"
kubectl -n rhacs-chs64i3dabr0026a6fag edit centrals.platform.stackrox.io test-central-1

# To run tests locally run:
make db/teardown db/setup db/migrate
make ocm/setup OCM_OFFLINE_TOKEN=<ocm-offline-token> OCM_ENV=development
make verify lint binary test test/integration

[vladbologa]

Could be exported here instead, so that we don't duplicate the annotation. Will do in a follow-up, if it's ok.

[SimonBaeumer]

Can you wait for @stehessel review?

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kurlov, SimonBaeumer, stehessel, vladbologa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [SimonBaeumer,kurlov,vladbologa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment