Custom resource metrics

[robbie-demuth]

What do you want to happen?

Prior to the Operator Framework's Operator SDK re-architecting itself as a Kubebuilder plugin, it contained libraries (since deleted in operator-framework/operator-sdk#3249) for generating and serving custom resource metrics (implemented using libraries from kube-state-metrics) and scaffolded operators to invoke said libraries on startup. As a result, operators built using the Operator SDK not only exposed the standard controller-runtime metrics, but also simple metrics about the existence of custom resources managed by the controller (i.e. <KIND>_info{<KIND>="<NAME>", namespace="<NAMESPACE>"} 1).

It'd be nice to provide these metrics for free via Kubebuilder's scaffolding or at least provide an easy way for operator developers to opt-in.

If possible, it'd also be nice to provide an easy, pluggable way for operator developers to declaratively instrument their operators with custom (adjective) custom resource metrics without having to do so in Reconcile (as documented by https://book.kubebuilder.io/reference/metrics.html#publishing-additional-metrics). For example, if a custom resource has a simple string field in its status subresource, it'd be nice to provide an API that allows operator developers to provide a metric name suffix, label name, and function for deriving label values from custom resources so that the controller generates and serves these metrics separately from Reconcile - preventing developers from having to instrument Reconcile wherever they update the status subresource's string field.

I wasn't sure whether to open this issue against kubebuilder or controller-runtime. I'm more than happy to move it to controller-runtime if desired. I also considered opening this issue against https://github.com/operator-framework/operator-lib since it's maintained as part of the Operator Framework. Alternatively, I could imagine the simple "info" metrics for custom resources being a feature request against kube-state-metrics. What do you all think?

Thanks!

Extra Labels

No response

[robbie-demuth]

RE my kube-state-metrics comment at the end, it looks like there have been some feature requests over there. The older of these two was closed because the maintainers felt that such metrics were the responsibility of operators. The other is still open and it looks like implementation is underway

kubernetes/kube-state-metrics#457
kubernetes/kube-state-metrics#1210

EDIT

It looks like the second of these two is focusing on custom resource definitions, not custom resources

[rashmigottipati]

@robbie-demuth Sounds interesting! Could you please describe some use cases on how you would use these standard controller runtime and metrics about existence of CRs?

[robbie-demuth]

The simple "info" metric without additional labels could be used to count custom resources by namespace, cluster, etc. Making additional labels or metrics pluggable, however, really opens up the door to anything

[rashmigottipati]

/cc @camilamacedo86

[camilamacedo86]

Hi @robbie-demuth,

Thank you to raise this issue.

Note that it was desirable in the past: #1068
I think that would be fit very well in a plugin to allow opt-in. See: https://book.kubebuilder.io/plugins/plugins.html
WDYT?

Please, feel free to contribute to this implementation. Your collab is very welcome.

[camilamacedo86]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[chrischdi]

Maybe worth to note that kube-state-metrics now supports custom resource definitions: kubernetes/kube-state-metrics#1710

Although it does not yet work for _info metrics: kubernetes/kube-state-metrics#1755

[camilamacedo86]

@Kavinjsir,

This one is related to the metrics so you might like to check it out. I think we need just to see how we could add the kube-state-metrics to kubebuilder.

[chrischdi]

Would be a great thing to have and at the Cluster API project would be interested in it too

• xref: Automate generation of kube-state-metrics configuration cluster-api#7158

I was about to ask the question in a new issue at kube-state-metrics if that is a reasonable way forward.

Happy to help working on it!

[Kavinjsir]

Sounds good! I'd be glad to work on this!

[Kavinjsir]

/assign

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[robbie-demuth]

kube-state-metrics now provides a declarative way to define custom resource state metrics, which our organization uses with great success, so I'm going to go ahead and close this

https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md