Update monitoring docs to use Prometheus and Grafana Operator

[dkwon17]

Is your enhancement related to a problem? Please describe

In the current monitoring docs, there are instructions to set up Prometheus and Grafana with Deployments and ConfigMaps.

Describe the solution you'd like

However, setting up the monitoring with the Prometheus Operator is a bit different because it requires using ServiceMonitor CRs to specify the metrics targets. It would be great if there were documentation for this.

For reference, the ServiceMonitors look similar to the following:

For monitoring Che Server:

- apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: che-host
  spec:
    endpoints:
    - interval: 30s
      port: metrics
      scheme: http
    selector:
      matchLabels:
        app.kubernetes.io/name: devspaces

For monitoring the DWO:

- apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: devworkspace-controller
  spec:
    endpoints:
    - interval: 30s
      port: metrics
      scheme: https
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      tlsConfig:
        insecureSkipVerify: true
    selector:
      matchLabels:
        app.kubernetes.io/name: devworkspace-controller

Describe alternatives you've considered

No response

Additional context

No response

[max-cx]

The docs team's downstream issue is https://issues.redhat.com/browse/RHDEVDOCS-4731.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.