Expose prometheus metrics from agent and operator

[invidian]

Original issues:

• agent: expose metrics coreos/container-linux-update-operator#157
• operator: expose metrics coreos/container-linux-update-operator#156

Do note, that some of this information can already be obtained using kube_node_labels metric:

kube_node_labels{container="kube-state-metrics",endpoint="http",job="kube-state-metrics",label_beta_kubernetes_io_arch="amd64",label_beta_kubernetes_io_os="linux",label_flatcar_linux_update_v1_flatcar_linux_net_group="alpha",label_flatcar_linux_update_v1_flatcar_linux_net_id="flatcar",label_flatcar_linux_update_v1_flatcar_linux_net_reboot_needed="false",label_flatcar_linux_update_v1_flatcar_linux_net_version="2671.0.0",label_foo="bar",label_kubernetes_io_arch="amd64",label_kubernetes_io_hostname="theory-main-workers-1",label_kubernetes_io_os="linux",namespace="prometheus-operator",node="theory-main-workers-1",pod="prometheus-operator-kube-state-metrics-d969c4b58-68zqw",service="prometheus-operator-kube-state-metrics"}

[HaveFun83]

Maybe there is also some inspiration at https://github.com/weaveworks/kured
Really cool will be some kind of "Blocking Reboots via Alerts" etc..

[invidian]

Maybe there is also some inspiration at https://github.com/weaveworks/kured

It seems they only expose single metric kured_reboot_required, but thanks!

Really cool will be some kind of "Blocking Reboots via Alerts" etc..

This seems like a new feature to me, not related to metrics. But thanks for the pointer too!

[jiphex]

fwiw I've written a small exporter to pull out some metrics from the updateengine package in this repository: it's here

[invidian]

@jiphex thanks for the contribution to the ecosystem, nice seeing community work around :)

I've seen your project yesterday actually. Please keep in mind though, that existing updateengine package has several bugs (although nothing which breaks functionality) and will have breaking changes in the next release (see commits in https://github.com/flatcar-linux/flatcar-linux-update-operator/commits/master/pkg/updateengine). And there should be one more breaking change done to the package, explained in #69.

There is also another copy of updateengine package in locksmith code. See #54. Ideally at some point all those would be unified.

[jiphex]

Thanks, I'll keep an eye out and update as required.

[invidian]

Related to this work, it would also be cool to have a Grafana dashboard consuming those metrics. E.g. I have a dashboard like this right now: