Merge pull request #29658 from kubernetes/dev-1.23

Official 1.23 Release Docs
kubernetes · Dec 7, 2021 · f0bb277 · f0bb277
2 parents aa2f69c + 03f82bf
commit f0bb277
Show file tree

Hide file tree

Showing 59 changed files with 46,232 additions and 856 deletions.
diff --git a/config.toml b/config.toml
@@ -139,10 +139,10 @@ time_format_default = "January 02, 2006 at 3:04 PM PST"
 description = "Production-Grade Container Orchestration"
 showedit = true
 
-latest = "v1.22"
+latest = "v1.23"
 
-fullversion = "v1.22.0"
-version = "v1.22"
+fullversion = "v1.23.0"
+version = "v1.23"
 githubbranch = "main"
 docsbranch = "main"
 deprecated = false
@@ -179,40 +179,40 @@ js = [
 ]
 
 [[params.versions]]
-fullversion = "v1.22.0"
-version = "v1.22"
-githubbranch = "v1.22.0"
+fullversion = "v1.23.0"
+version = "v1.23"
+githubbranch = "v1.23.0"
 docsbranch = "main"
 url = "https://kubernetes.io"
 
 [[params.versions]]
-fullversion = "v1.21.4"
+fullversion = "v1.22.4"
+version = "v1.22"
+githubbranch = "v1.22.4"
+docsbranch = "release-1.22"
+url = "https://v1-22.docs.kubernetes.io"
+
+[[params.versions]]
+fullversion = "v1.21.7"
 version = "v1.21"
-githubbranch = "v1.21.4"
+githubbranch = "v1.21.7"
 docsbranch = "release-1.21"
 url = "https://v1-21.docs.kubernetes.io"
 
 [[params.versions]]
-fullversion = "v1.20.10"
+fullversion = "v1.20.13"
 version = "v1.20"
-githubbranch = "v1.20.10"
+githubbranch = "v1.20.13"
 docsbranch = "release-1.20"
 url = "https://v1-20.docs.kubernetes.io"
 
 [[params.versions]]
-fullversion = "v1.19.14"
+fullversion = "v1.19.16"
 version = "v1.19"
-githubbranch = "v1.19.14"
+githubbranch = "v1.19.16"
 docsbranch = "release-1.19"
 url = "https://v1-19.docs.kubernetes.io"
 
-[[params.versions]]
-fullversion = "v1.18.20"
-version = "v1.18"
-githubbranch = "v1.18.20"
-docsbranch = "release-1.18"
-url = "https://v1-18.docs.kubernetes.io"
-
 # User interface configuration
 [params.ui]
 # Enable to show the side bar menu in its compact state.

diff --git a/content/en/blog/_posts/2018-10-01-health-checking-grpc.md b/content/en/blog/_posts/2018-10-01-health-checking-grpc.md
@@ -4,6 +4,8 @@ title:  'Health checking gRPC servers on Kubernetes'
 date: 2018-10-01
 ---
 
+_Built-in gRPC probes were introduced in Kubernetes 1.23. To learn more, see [Configure Liveness, Readiness and Startup Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe)._
+
 **Author**: [Ahmet Alp Balkan](https://twitter.com/ahmetb) (Google)
 
 [gRPC](https://grpc.io) is on its way to becoming the lingua franca for

diff --git a/content/en/docs/concepts/architecture/cri.md b/content/en/docs/concepts/architecture/cri.md
@@ -0,0 +1,51 @@
+---
+title: Container Runtime Interface (CRI)
+content_type: concept
+weight: 50
+---
+
+<!-- overview -->
+
+The CRI is a plugin interface which enables the kubelet to use a wide variety of
+container runtimes, without having a need to recompile the cluster components.
+
+You need a working
+{{<glossary_tooltip text="container runtime" term_id="container-runtime">}} on
+each Node in your cluster, so that the
+{{< glossary_tooltip text="kubelet" term_id="kubelet" >}} can launch
+{{< glossary_tooltip text="Pods" term_id="pod" >}} and their containers.
+
+{{< glossary_definition term_id="container-runtime-interface" length="all" >}}
+
+<!-- body -->
+
+## The API {#api}
+
+{{< feature-state for_k8s_version="v1.23" state="stable" >}}
+
+The kubelet acts as a client when connecting to the container runtime via gRPC.
+The runtime and image service endpoints have to be available in the container
+runtime, which can be configured separately within the kubelet by using the
+`--image-service-endpoint` and `--container-runtime-endpoint` [command line
+flags](/docs/reference/command-line-tools-reference/kubelet)
+
+For Kubernetes v{{< skew currentVersion >}}, the kubelet prefers to use CRI `v1`.
+If a container runtime does not support `v1` of the CRI, then the kubelet tries to
+negotiate any older supported version.
+The v{{< skew currentVersion >}} kubelet can also negotiate CRI `v1alpha2`, but
+this version is considered as deprecated.
+If the kubelet cannot negotiate a supported CRI version, the kubelet gives up
+and doesn't register as a node.
+
+## Upgrading
+
+When upgrading Kubernetes, then the kubelet tries to automatically select the
+latest CRI version on restart of the component. If that fails, then the fallback
+will take place as mentioned above. If a gRPC re-dial was required because the
+container runtime has been upgraded, then the container runtime must also
+support the initially selected version or the redial is expected to fail. This
+requires a restart of the kubelet.
+
+## {{% heading "whatsnext" %}}
+
+- Learn more about the CRI [protocol definition](https://github.com/kubernetes/cri-api/blob/c75ef5b/pkg/apis/runtime/v1/api.proto)
diff --git a/content/en/docs/concepts/architecture/nodes.md b/content/en/docs/concepts/architecture/nodes.md
@@ -424,20 +424,104 @@ for gracefully terminating normal pods, and the last 10 seconds would be
 reserved for terminating [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical).
 
 {{< note >}}
-When pods were evicted during the graceful node shutdown, they are marked as failed.
-Running `kubectl get pods` shows the status of the the evicted pods as `Shutdown`.
+When pods were evicted during the graceful node shutdown, they are marked as shutdown.
+Running `kubectl get pods` shows the status of the the evicted pods as `Terminated`.
 And `kubectl describe pod` indicates that the pod was evicted because of node shutdown:
 
 ```
-Status:         Failed
-Reason:         Shutdown
-Message:        Node is shutting, evicting pods
+Reason:         Terminated
+Message:        Pod was terminated in response to imminent node shutdown.
 ```
 
-Failed pod objects will be preserved until explicitly deleted or [cleaned up by the GC](/docs/concepts/workloads/pods/pod-lifecycle/#pod-garbage-collection).
-This is a change of behavior compared to abrupt node termination.
 {{< /note >}}
 
+### Pod Priority based graceful node shutdown {#pod-priority-graceful-node-shutdown}
+
+{{< feature-state state="alpha" for_k8s_version="v1.23" >}}
+
+To provide more flexibility during graceful node shutdown around the ordering
+of pods during shutdown, graceful node shutdown honors the PriorityClass for
+Pods, provided that you enabled this feature in your cluster. The feature
+allows allows cluster administers to explicitly define the ordering of pods
+during graceful node shutdown based on [priority
+classes](docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass).
+
+The [Graceful Node Shutdown](#graceful-node-shutdown) feature, as described
+above, shuts down pods in two phases, non-critical pods, followed by critical
+pods. If additional flexibility is needed to explicitly define the ordering of
+pods during shutdown in a more granular way, pod priority based graceful
+shutdown can be used.
+
+When graceful node shutdown honors pod priorities, this makes it possible to do
+graceful node shutdown in multiple phases, each phase shutting down a
+particular priority class of pods. The kubelet can be configured with the exact
+phases and shutdown time per phase.
+
+Assuming the following custom pod [priority
+classes](docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass)
+in a cluster,
+
+|Pod priority class name|Pod priority class value|
+|-------------------------|------------------------|
+|`custom-class-a`         | 100000                 |
+|`custom-class-b`         | 10000                  |
+|`custom-class-c`         | 1000                   |
+|`regular/unset`          | 0                      |
+
+Within the [kubelet configuration](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)
+the settings for `shutdownGracePeriodByPodPriority` could look like:
+
+|Pod priority class value|Shutdown period|
+|------------------------|---------------|
+| 100000                 |10 seconds     |
+| 10000                  |180 seconds    |
+| 1000                   |120 seconds    |
+| 0                      |60 seconds     |
+
+The corresponding kubelet config YAML configuration would be:
+
+```yaml
+shutdownGracePeriodByPodPriority:
+  - priority: 100000
+    shutdownGracePeriodSeconds: 10
+  - priority: 10000
+    shutdownGracePeriodSeconds: 180
+  - priority: 1000
+    shutdownGracePeriodSeconds: 120
+  - priority: 0
+    shutdownGracePeriodSeconds: 60
+```
+
+The above table implies that any pod with priority value >= 100000 will get
+just 10 seconds to stop, any pod with value >= 10000 and < 100000 will get 180
+seconds to stop, any pod with value >= 1000 and < 10000 will get 120 seconds to stop.
+Finally, all other pods will get 60 seconds to stop.
+
+One doesn't have to specify values corresponding to all of the classes. For
+example, you could instead use these settings:
+
+|Pod priority class value|Shutdown period|
+|------------------------|---------------|
+| 100000                 |300 seconds    |
+| 1000                   |120 seconds    |
+| 0                      |60 seconds     |
+
+
+In the above case, the pods with custom-class-b will go into the same bucket
+as custom-class-c for shutdown.
+
+If there are no pods in a particular range, then the kubelet does not wait
+for pods in that priority range. Instead, the kubelet immediately skips to the
+next priority class value range.
+
+If this feature is enabled and no configuration is provided, then no ordering
+action will be taken.
+
+Using this feature, requires enabling the
+`GracefulNodeShutdownBasedOnPodPriority` feature gate, and setting the kubelet
+config's `ShutdownGracePeriodByPodPriority` to the desired configuration
+containing the pod priority class values and their respective shutdown periods.
+
 ## Swap memory management {#swap-memory}
 
 {{< feature-state state="alpha" for_k8s_version="v1.22" >}}

diff --git a/content/en/docs/concepts/cluster-administration/networking.md b/content/en/docs/concepts/cluster-administration/networking.md
@@ -64,7 +64,7 @@ This means that containers within a `Pod` can all reach each other's ports on
 usage, but this is no different from processes in a VM.  This is called the
 "IP-per-pod" model.
 
-How this is implemented is a detail of the particular container runtime in use.
+How this is implemented is a detail of the particular container runtime in use. Likewise, the networking option you choose may support [dual-stack IPv4/IPv6 networking](/docs/concepts/services-networking/dual-stack/); implementations vary.
 
 It is possible to request ports on the `Node` itself which forward to your `Pod`
 (called host ports), but this is a very niche operation. How that forwarding is

diff --git a/content/en/docs/concepts/cluster-administration/system-logs.md b/content/en/docs/concepts/cluster-administration/system-logs.md
@@ -22,14 +22,62 @@ generates log messages for the Kubernetes system components.
 
 For more information about klog configuration, see the [Command line tool reference](/docs/reference/command-line-tools-reference/).
 
-An example of the klog native format:
+Kubernetes is in the process of simplifying logging in its components. The
+following klog command line flags [are
+deprecated](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
+starting with Kubernetes 1.23 and will be removed in a future release:
+
+- `--add-dir-header`
+- `--alsologtostderr`
+- `--log-backtrace-at`
+- `--log-dir`
+- `--log-file`
+- `--log-file-max-size`
+- `--logtostderr`
+- `--one-output`
+- `--skip-headers`
+- `--skip-log-headers`
+- `--stderrthreshold`
+
+Output will always be written to stderr, regardless of the output
+format. Output redirection is expected to be handled by the component which
+invokes a Kubernetes component. This can be a POSIX shell or a tool like
+systemd.
+
+In some cases, for example a distroless container or a Windows system service,
+those options are not available. Then the
+[`kube-log-runner`](https://github.com/kubernetes/kubernetes/blob/d2a8a81639fcff8d1221b900f66d28361a170654/staging/src/k8s.io/component-base/logs/kube-log-runner/README.md)
+binary can be used as wrapper around a Kubernetes component to redirect
+output. A prebuilt binary is included in several Kubernetes base images under
+its traditional name as `/go-runner` and as `kube-log-runner` in server and
+node release archives.
+
+This table shows how `kube-log-runner` invocations correspond to shell redirection:
+
+| Usage                                    | POSIX shell (such as bash) | `kube-log-runner <options> <cmd>`                           |
+| -----------------------------------------|----------------------------|-------------------------------------------------------------|
+| Merge stderr and stdout, write to stdout | `2>&1`                     | `kube-log-runner` (default behavior)                        |
+| Redirect both into log file              | `1>>/tmp/log 2>&1`         | `kube-log-runner -log-file=/tmp/log`                        |
+| Copy into log file and to stdout         | `2>&1 \| tee -a /tmp/log`  | `kube-log-runner -log-file=/tmp/log -also-stdout`           |
+| Redirect only stdout into log file       | `>/tmp/log`                | `kube-log-runner -log-file=/tmp/log -redirect-stderr=false` |
+
+### Klog output
+
+An example of the traditional klog native format:
 ```
 I1025 00:15:15.525108       1 httplog.go:79] GET /api/v1/namespaces/kube-system/pods/metrics-server-v0.3.1-57c75779f-9p8wg: (1.512ms) 200 [pod_nanny/v0.0.0 (linux/amd64) kubernetes/$Format 10.56.1.19:51756]
 ```
 
+The message string may contain line breaks:
+```
+I1025 00:15:15.525108       1 example.go:79] This is a message
+which has a line break.
+```
+
+
 ### Structured Logging
 
-{{< feature-state for_k8s_version="v1.19" state="alpha" >}}
+{{< feature-state for_k8s_version="v1.23" state="beta" >}}
 
 {{< warning >}}
 Migration to structured log messages is an ongoing process. Not all log messages are structured in this version. When parsing log files, you must also handle unstructured log messages.
@@ -38,9 +86,11 @@ Log formatting and value serialization are subject to change.
 {{< /warning>}}
 
 Structured logging introduces a uniform structure in log messages allowing for programmatic extraction of information. You can store and process structured logs with less effort and cost.
-New message format is backward compatible and enabled by default.
+The code which generates a log message determines whether it uses the traditional unstructured klog output
+or structured logging.
 
-Format of structured logs:
+The default formatting of structured log messages is as text, with a format that
+is backward compatible with traditional klog:
 
 ```ini
 <klog header> "<message>" <key1>="<value1>" <key2>="<value2>" ...
@@ -52,6 +102,13 @@ Example:
 I1025 00:15:15.525108       1 controller_utils.go:116] "Pod status updated" pod="kube-system/kubedns" status="ready"
 ```
 
+Strings are quoted. Other values are formatted with
+[`%+v`](https://pkg.go.dev/fmt#hdr-Printing), which may cause log messages to
+continue on the next line [depending on the data](https://github.com/kubernetes/kubernetes/issues/106428).
+```
+I1025 00:15:15.525108       1 example.go:116] "Example" data="This is text with a line break\nand \"quotation marks\"." someInt=1 someFloat=0.1 someStruct={StringField: First line,
+second line.}
+```
 
 ### JSON log format
 
@@ -82,7 +139,7 @@ Example of JSON log format (pretty printed):
 
 Keys with special meaning:
 * `ts` - timestamp as Unix time (required, float)
-* `v` - verbosity (required, int, default 0)
+* `v` - verbosity (only for info and not for error messages, int)
 * `err` - error string (optional, string)
 * `msg` - message (required, string)
 
@@ -139,4 +196,5 @@ The `logrotate` tool rotates logs daily, or once the log size is greater than 10
 
 * Read about the [Kubernetes Logging Architecture](/docs/concepts/cluster-administration/logging/)
 * Read about [Structured Logging](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1602-structured-logging)
+* Read about [deprecation of klog flags](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
 * Read about the [Conventions for logging severity](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md)
diff --git a/content/en/docs/concepts/extend-kubernetes/_index.md b/content/en/docs/concepts/extend-kubernetes/_index.md
@@ -77,7 +77,7 @@ failure.
 In the webhook model, Kubernetes makes a network request to a remote service.
 In the *Binary Plugin* model, Kubernetes executes a binary (program).
 Binary plugins are used by the kubelet (e.g.
-[Flex Volume Plugins](/docs/concepts/storage/volumes/#flexVolume)
+[Flex Volume Plugins](/docs/concepts/storage/volumes/#flexvolume)
 and [Network Plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/))
 and by kubectl.
 
@@ -163,6 +163,8 @@ After a request is authorized, if it is a write operation, it also goes through
 ) allow users to mount volume types without built-in support by having the
 Kubelet call a Binary Plugin to mount the volume.
 
+FlexVolume is deprecated since Kubernetes v1.23. The Out-of-tree CSI driver is the recommended way to write volume drivers in Kubernetes. See [Kubernetes Volume Plugin FAQ for Storage Vendors](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md#kubernetes-volume-plugin-faq-for-storage-vendors) for more information.
+
 
 ### Device Plugins