Document setting up prometheus with an operator created cluster

[chrisseto]

I've created an CockroachDB cluster via the operator and stumbled through settings up prometheus monitoring by following the tutorial in the CockroachDB repo.

The link to the prometheus operator manifest in the above tutorial appears to have expired. I tracked down the bundle in the prometheus operator repo.

Everything else appears to work just fine with the exception of the ServiceMonitor which points at an incorrect CA.

I was able to get it working by swapping the tlsConfig to the below value. The config needs access to the ca.crt which is not present in the cockroachdb-ca secret so I fell back to using the cockroachdb-node secret. We should likely include the ca.crt in cockroachdb-ca or in a separate secret for ease of use before fully documenting this.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: cockroachdb
    prometheus: cockroachdb
  name: cockroachdb
spec:
  endpoints:
  - path: /_status/vars
    port: http
    tlsConfig:
      ca:
        secret:
          key: ca.crt
          name: cockroachdb-node
      serverName: 127.0.0.1
  selector:
    matchLabels:
      prometheus: cockroachdb

cc @chudro

[alinadonisa]

@keith-mcclellan, @johnrk we should open another issue for moving the ca.crt in the ca secret

[johnrk-zz]

@alinadonisa , can you create this new issue with the steps that you had in mind?

[keith-mcclellan]

the cockroach-ca secret is just for minting new certs as it has the private key, prometheus just needs the public key for the ca so it can connect to the prometheus endpoint since it's not an authenticated interaction. What we need to do is have the ca.crt live in it's own secret so it can be mounted independently of the node.crt it sounds like. Is that a correct summary @chrisseto ?

[chrisseto]

An independent secret with the ca.crt would work just as well 👍

[alinadonisa]

@johnrk, @keith-mcclellan, @chrisseto I added issue #491 for this implementation.

[johnrk-zz]

@alinadonisa , thank you!