feat(kuma-cp) metrics

[jakubdyszkiewicz]

Summary

This PR introduces Control Plane metrics:

• Latencies and response codes etc. for API Server/Admin Server/Bootstrap Server/DNS Server/SDS/XDS/KDS
• XDS: summary of XDS generation (time, count)
• XDS: active connections
• SDS: summary of SDS generation (time, count)
• SDS: cert generations
• KDS: summary of KDS generation (time, count)
• KDS: client-side stats
• Store: latencies of underlying storage
• Store cache: number of hits and misses for cache
• Static Info about the CP
• Leader election
• Go (GC, threads etc) and process info

It does not include dashboards.

Implementation

My first approach was to use promauto with global default Prometheus registry and MustRegister that panic, but it was a disaster in tests, therefore I implicitly pass the registry in the Metric object.

I try to register metrics as "high" in Setup/component.go etc. as possible. I was trying to avoid spreading Prometheus code across the codebase.

For latencies, I try to use a Summary, not Histogram. You can read about the differences here https://prometheus.io/docs/practices/histograms/ as long as there is no aggregation, Histograms are just easier to use IMHO.

Documentation

• todo After I introduce dashboards

[nickolaev]

V(1)? we need it?

[jakubdyszkiewicz]

I think it's overkill to log every DNS request on the info level. It's like with API Server, we don't request every single request. With many services that uses DNS, logs will be spammed with "Query for..."