diff --git a/content/en/docs/concepts/cluster-administration/monitoring.md b/content/en/docs/concepts/cluster-administration/monitoring.md new file mode 100644 index 0000000000000..607bb7c1a1347 --- /dev/null +++ b/content/en/docs/concepts/cluster-administration/monitoring.md @@ -0,0 +1,105 @@ +--- +title: Metrics For The Kubernetes Control Plane +reviewers: +- brancz +- logicalhan +- RainbowMango +content_template: templates/concept +weight: 60 +--- + +{{% capture overview %}} + +System component metrics can give a better look into what is happening inside them. Metrics are particularly useful for building dashboards and alerts. + +Metrics in Kubernetes control plane components are exposed in Prometheus text format. + +{{% /capture %}} + +{{% capture body %}} + +## Metrics in Kubernetes + +In most cases those metrics are available on `/metrics` endpoint of the HTTP server. For components that doesn't expose endpoint by default it can be enabled using `--bind-address` flag. + +Examples of those components: +* {{< glossary_tooltip term_id="kube-controller-manager" text="kube-controller-manager" >}} +* {{< glossary_tooltip term_id="kube-proxy" text="kube-proxy" >}} +* {{< glossary_tooltip term_id="kube-apiserver" text="kube-apiserver" >}} +* {{< glossary_tooltip term_id="kube-scheduler" text="kube-scheduler" >}} +* {{< glossary_tooltip term_id="kubelet" text="kubelet" >}} + +Note that {{< glossary_tooltip term_id="kubelet" text="kubelet" >}} also exposes metrics in `/metrics/cadvisor` and `/metrics/resource` endpoints. Those metrics do not have same lifecycle. + +If your cluster uses {{< glossary_tooltip term_id="rbac" text="RBAC" >}}, reading metrics requires authorization via a user, group or ServiceAccount with a ClusterRole that allows accessing `/metrics`. +For example: +``` +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: prometheus +rules: + - nonResourceURLs: + - "/metrics" + verbs: + - get +``` + +## Metric lifecycle + +Alpha metric → Stable metric → Deprecated metric → Hidden metric → Deletion + +Alpha metrics have no stability guarantees; as such they can be modified or deleted at any time. + +Stable metrics can be guaranteed to not change; Specifically, stability means: + +* the metric itself will not be deleted (or renamed) +* the type of metric will not be modified +* no labels can be added or removed from this metric + +Deprecated metric signal that the metric will eventually be deleted; to find which version, you need to check annotation, which includes from which kubernetes version that metric will be considered deprecated. + +Before deprecation: + +``` +# HELP some_counter this counts things +# TYPE some_counter counter +some_counter 0 +``` + +After deprecation: + +``` +# HELP some_counter (Deprecated since 1.15.0) this counts things +# TYPE some_counter counter +some_counter 0 +``` + +Hidden metrics will no longer be exposed by default; to use a hidden metric, you need to override the configuration for the relevant cluster component. + +Deleted metrics will no longer be available. + + +## Show Hidden Metrics + +As described above, admins can enable hidden metrics through a command-line flag on a specific binary. This intends to be used as an escape hatch for admins if they missed the migration of the metrics deprecated in the last release. + +The flag `show-hidden-metrics-for-version` takes a version for which you want to show metrics deprecated in that release. The version is expressed as x.y, where x is the major version, y is the minor version. The patch version is not needed even though a metrics can be deprecated in a patch release, the reason for that is the metrics deprecation policy runs against the minor release. + +The flag can only take the previous minor version as it's value. All metrics hidden in previous will be emitted if admins set the previous version to `show-hidden-metrics-for-version`. The too old version is not allowed because this violates the metrics deprecated policy. + +Take metric `A` as an example, here assumed that `A` is deprecated in 1.n. According to metrics deprecated policy, we can reach the following conclusion: + +* In release `1.n`, the metric is deprecated, and it can be emitted by default. +* In release `1.n+1`, the metric is hidden by default and it can be emitted by command line `show-hidden-metrics-for-version=1.n`. +* In release `1.n+2`, the metric should be removed from the codebase. No escape hatch anymore. + +If you're upgrading from release `1.12` to `1.13`, but still depend on a metric `A` deprecated in `1.12`, you should set hidden metrics via command line: `--show-hidden-metrics=1.12` and remember to remove this metric dependency before upgrading to `1.14` + +{{% /capture %}} + +{{% capture whatsnext %}} +* Read about the [Prometheus text format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format) for metrics +* See the list of [stable Kubernetes metrics](https://github.com/kubernetes/kubernetes/blob/master/test/instrumentation/testdata/stable-metrics-list.yaml) +* Read about the [Kubernetes deprecation policy](https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-a-feature-or-behavior ) +{{% /capture %}} diff --git a/data/concepts.yml b/data/concepts.yml index 998b265e0666f..a20948c8abdd3 100644 --- a/data/concepts.yml +++ b/data/concepts.yml @@ -116,6 +116,7 @@ toc: - docs/concepts/cluster-administration/networking.md - docs/concepts/cluster-administration/network-plugins.md - docs/concepts/cluster-administration/logging.md + - docs/concepts/cluster-administration/monitoring.md - docs/concepts/cluster-administration/kubelet-garbage-collection.md - docs/concepts/cluster-administration/federation.md - docs/concepts/cluster-administration/sysctl-cluster.md