Expose metrics for different actions performed during reconciliation in CassandraDatacenter controller

[jsanda]

What is missing?
The controller-runtime already exposes a Prometheus for us that provides several metrics like total number of reconciliations per controller and the duration of reconciliations. We need to expose additional metrics for various operations that the controller performs. See #215 for an example.

Note: Part of the work involved for this issue is to determine what metrics we want to expose. This may involve creating additional, follow up tickets.

As starting point I think we should expose metrics for any actions that involve management-api calls.

We also have multiple places where we perform DC-wide actions that involve iterating over all pods and usually calls to the management-api as well. Two examples are starting C* nodes (see here) and reconciling full query logging (see here). It would be good to get metrics for those DC-wide operations.

Why do we need it?
We need observability for what cass-operator is actually doing. This will also give us better observability around the management-api and Cassandra itself.

Environment

• Cass Operator version:

  v1.11.0

  **Anything else we need to know?**:

┆Issue is synchronized with this Jira Story by Unito
┆Issue Number: CASS-38