Document control plane monitoring

content/en/docs/concepts/cluster-administration/monitoring.md

@@ -0,0 +1,103 @@
+---
+title: Monitoring Control Plane Components
+reviewers:
+- brancz
+- logicalhan
+- RainbowMango
+content_template: templates/concept
+weight: 60
+---
+
+{{% capture overview %}}
+
+System component metrics can give a better look into what is happening inside them. Metrics are particularly useful for building dashboards and alerts.
+
+{{% /capture %}}
+
+{{% capture body %}}
+
+## Metrics in Kubernetes
+
+Kubernetes control plane components publish metrics in Prometheus text format. In most cases those metrics are available on `/metrics` endpoint of the HTTP server. For components that doesn't expose endpoint by default it can be enabled using `--bind-address` flag.
+
+Examples of those components:
+* {{< glossary_tooltip term_id="kube-controller-manager" text="kube-controller-manager" >}}
+* {{< glossary_tooltip term_id="kube-proxy" text="kube-proxy" >}}
+* {{< glossary_tooltip term_id="kube-apiserver" text="kube-apiserver" >}}
+* {{< glossary_tooltip term_id="kube-scheduler" text="kube-scheduler" >}}
+* kube-proxy
+* kube-apiserver
+* kube-scheduler
+* kubelet (metrics exposed on `/metrics/cadvisor` and `/metrics/resource` do not have same guarantee)
+
+If your cluster uses {{< glossary_tooltip term_id="rbac" text="RBAC" >}}, reading metrics requires authorization via a user, group or ServiceAccount with a ClusterRole that allows accessing `/metrics`.
+For example:
+```
+apiVersion: rbac.authorization.k8s.io/v1	
+kind: ClusterRole	
+metadata:	
+  name: prometheus	
+rules:	
+  - nonResourceURLs:	
+      - "/metrics"	
+    verbs:	
+      - get
+```
+
+## Metric lifecycle
+
+Alpha metric →  Stable metric →  Deprecated metric →  Hidden metric → Deletion
+
+Alpha metrics have no stability guarantees; as such they can be modified or deleted at any time.
+
+Stable metrics can be guaranteed to not change; Specifically, stability means:
+
+* the metric itself will not be deleted (or renamed)
+* the type of metric will not be modified
+* no labels can be added or removed from this metric
+
+Deprecated metric signal that the metric will eventually be deleted; to find which version, you need to check annotation, which includes from which kubernetes version that metric will be considered deprecated.
+
+Before deprecation:
+
+```
+# HELP some_counter this counts things
+# TYPE some_counter counter
+some_counter 0
+```
+
+After deprecation:
+
+```
+# HELP some_counter (Deprecated from 1.15) this counts things
+# TYPE some_counter counter
+some_counter 0
+```
+
+Hidden metrics will no longer be exposed by default; to use a hidden metric, you need to override the configuration for the relevant cluster component.
+
+Deleted metrics will no longer be available.
+
+
+## Show Hidden Metrics
+
+As described above, admins can enable hidden metrics through a command-line flag on a specific binary. This intends to be used as an escape hatch for admins if they missed the migration of the metrics deprecated in the last release.
+
+The flag `show-hidden-metrics-for-version` takes a version for which you want to show metrics deprecated in that release. The version is expressed as x.y, where x is the major version, y is the minor version. The patch version is not needed even though a metrics can be deprecated in a patch release, the reason for that is the metrics deprecation policy runs against the minor release.
+
+The flag can only take the previous minor version as it's value. All metrics hidden in previous will be emitted if admins set the previous version to `show-hidden-metrics-for-version`. The too old version is not allowed because this violates the metrics deprecated policy.
+
+Take metric `A` as an example, here assumed that `A` is deprecated in 1.n. According to metrics deprecated policy, we can reach the following conclusion:
+
+* In release `1.n`, the metric is deprecated, and it can be emitted by default.
+* In release `1.n+1`, the metric is hidden by default and it can be emitted by command line `show-hidden-metrics-for-version=1.n`.
+* In release `1.n+2`, the metric should be removed from the codebase. No escape hatch anymore.
+
+So, if admins want to enable metric `A` in release `1.n+1`, they should set `1.n` to the command line flag. That is `show-hidden-metrics=1.n`.
+
+{{% /capture %}}
+
+{{% capture whatsnext %}}
+* Read about the [Prometheus text format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format) for metrics
+* See the list of [stable Kubernetes metrics](https://github.com/kubernetes/kubernetes/blob/master/test/instrumentation/testdata/stable-metrics-list.yaml)
+{{% /capture %}}

data/concepts.yml

@@ -116,6 +116,7 @@ toc:
   - docs/concepts/cluster-administration/networking.md
   - docs/concepts/cluster-administration/network-plugins.md
   - docs/concepts/cluster-administration/logging.md
+  - docs/concepts/cluster-administration/monitoring.md
   - docs/concepts/cluster-administration/kubelet-garbage-collection.md
   - docs/concepts/cluster-administration/federation.md
   - docs/concepts/cluster-administration/sysctl-cluster.md