Expose MR Metrics

[humoflife]

What problem are you facing?

Crossplane users would benefit from the ability to offer service level objectives (SLOs) to their users. To do this they need MR related metrics.

How could Crossplane help solve your problem?

At the Crossplane pod port 8080/metrics interface, expose the following:

• number of MRs total, by claim, by composition

• number of MRs in Synched False State total, by claim, by composition

• number of MRs in Synched True State total, by claim, by composition

• number of MRs in Synched Unkown State total, by claim, by composition

• number of MRs in Ready False State total, by claim, by composition

• number of MRs in Ready True State total, by claim, by composition

• number of MRs in Ready Unknown State total, by claim, by composition

• number of MRs in Synched True and Ready True State total, by claim, by composition

• number of MRs in Synched False and Ready True State total, by claim, by composition

• number of MRs in Synched Unknown and Ready True State total, by claim, by composition

• number of MRs in Synched True and Ready False State total, by claim, by composition

• number of MRs in Synched False and Ready False State total, by claim, by composition

• number of MRs in Synched Unknown and Ready False State total, by claim, by composition

• for each MR, time to readiness (Synched True -> Ready True)

• for each MR, time to non-readiness (Ready True -> Synched !True)

• for each MR, number of Synched state changes

• for each MR, number of Ready state changes

• for each MR, time from deletion request to external resource and MR deletion

• for each claim, time to readiness (from claim created to ready)

• for each claim, time to delete all associated resources (from claim to external resources removed)

• if possible, control plane availability and uptime (since last Crossplane pod restart).

The implementation of the metric gathering could be performed in the Crossplane runtime reconciler code, line 675+.
https://github.com/crossplane/crossplane-runtime/blob/master/pkg/reconciler/managed/reconciler.go#L675

An example time to readiness calculation is located in Uptest at https://github.com/upbound/uptest/blob/6e567ebd9ed30f1b1670d2cbbb679fde9beebc6b/cmd/perf/internal/managed/managed.go#L171

[negz]

Something to double-check for whoever works on this: do we need to add all of these metrics as individual time series? Let's make sure we focus on adding low-level, flexible time series that can be composed to produce useful higher level metrics.

[blut]

For anyone working with crossplane https://github.com/crossplane-contrib/x-metrics might provide useful metrics to aggregate the requested metrics.

[pierluigilenoci]

@ezgidemirel, from your experience and knowledge, how much hope is there to see this or something similar implemented in Crossplane? 🙏🏻

Thank you. ❤️

[momoXD007]

From an observability point of view Crossplane so far is lacking a good strategy to monitor the MRs and claims in "kubernetes native way".
It is hard to track the overall health of an environment without understanding if just one single claim or MR is not synched or suddenly hundreds of them are not synched.

It is currently also hard to understand adoption of Crossplane in a bigger environment: if you can't pinpoint which compositions are used the most.

[ezgidemirel]

Hi @pierluigilenoci, as we discussed in the slack thread, I'm closing this issue as completed.

We have introduced MR metrics with #683 but decided not to add MR names, claim names or composite names to the exposed metrics as labels. The reason is, we don't want to create a metric for each managed resource created on the cluster. This will increase the cardinality dramatically. You can see more details in the comment here.

Further discussions about having more detailed metrics can be carried on within this issue.