logging: structured logging, klog deprecation

This is primarily the docs PR for
- alpha: kubernetes/enhancements#2845
- beta: kubernetes/enhancements#1602

But as it touches the file, it also updates the examples and explanations to
reflect some recent observations.

content/en/docs/concepts/cluster-administration/system-logs.md

@@ -22,14 +22,62 @@ generates log messages for the Kubernetes system components.
 
 For more information about klog configuration, see the [Command line tool reference](/docs/reference/command-line-tools-reference/).
 
-An example of the klog native format:
+Kubernetes is in the process of simplifying logging in its components. The
+following klog command line flags [are
+deprecated](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
+starting with Kubernetes 1.23 and will be removed in a future release:
+
+- `--add-dir-header`
+- `--alsologtostderr`
+- `--log-backtrace-at`
+- `--log-dir`
+- `--log-file`
+- `--log-file-max-size`
+- `--logtostderr`
+- `--one-output`
+- `--skip-headers`
+- `--skip-log-headers`
+- `--stderrthreshold`
+
+Output will always be written to stderr, regardless of the output
+format. Output redirection is expected to be handled by the component which
+invokes a Kubernetes component. This can be a POSIX shell or a tool like
+systemd.
+
+In some cases, for example a distroless container or a Windows system service,
+those options are not available. Then the
+[`kube-log-runner`](https://github.com/kubernetes/kubernetes/blob/d2a8a81639fcff8d1221b900f66d28361a170654/staging/src/k8s.io/component-base/logs/kube-log-runner/README.md)
+binary can be used as wrapper around a Kubernetes component to redirect
+output. A prebuilt binary is included in several Kubernetes base images under
+its traditional name as `/go-runner` and as `kube-log-runner` in server and
+node release archives.
+
+This table shows how `kube-log-runner` invocations correspond to shell redirection:
+
+| Usage                                    | POSIX shell (such as bash) | `kube-log-runner <options> <cmd>`                           |
+| -----------------------------------------|----------------------------|-------------------------------------------------------------|
+| Merge stderr and stdout, write to stdout | `2>&1`                     | `kube-log-runner` (default behavior)                        |
+| Redirect both into log file              | `1>>/tmp/log 2>&1`         | `kube-log-runner -log-file=/tmp/log`                        |
+| Copy into log file and to stdout         | `2>&1 \| tee -a /tmp/log`  | `kube-log-runner -log-file=/tmp/log -also-stdout`           |
+| Redirect only stdout into log file       | `>/tmp/log`                | `kube-log-runner -log-file=/tmp/log -redirect-stderr=false` |
+
+### Klog output
+
+An example of the traditional klog native format:
 ```
 I1025 00:15:15.525108       1 httplog.go:79] GET /api/v1/namespaces/kube-system/pods/metrics-server-v0.3.1-57c75779f-9p8wg: (1.512ms) 200 [pod_nanny/v0.0.0 (linux/amd64) kubernetes/$Format 10.56.1.19:51756]
 ```
 
+The message string may contain line breaks:
+```
+I1025 00:15:15.525108       1 example.go:79] This is a message
+which has a line break.
+```
+
+
 ### Structured Logging
 
-{{< feature-state for_k8s_version="v1.19" state="alpha" >}}
+{{< feature-state for_k8s_version="v1.23" state="beta" >}}
 
 {{< warning >}}
 Migration to structured log messages is an ongoing process. Not all log messages are structured in this version. When parsing log files, you must also handle unstructured log messages.
@@ -38,9 +86,11 @@ Log formatting and value serialization are subject to change.
 {{< /warning>}}
 
 Structured logging introduces a uniform structure in log messages allowing for programmatic extraction of information. You can store and process structured logs with less effort and cost.
-New message format is backward compatible and enabled by default.
+The code which generates a log message determines whether it uses the traditional unstructured klog output
+or structured logging.
 
-Format of structured logs:
+The default formatting of structured log messages is as text, with a format that
+is backward compatible with traditional klog:
 
 ```ini
 <klog header> "<message>" <key1>="<value1>" <key2>="<value2>" ...
@@ -52,6 +102,13 @@ Example:
 I1025 00:15:15.525108       1 controller_utils.go:116] "Pod status updated" pod="kube-system/kubedns" status="ready"
 ```
 
+Strings are quoted. Other values are formatted with
+[`%+v`](https://pkg.go.dev/fmt#hdr-Printing), which may cause log messages to
+continue on the next line [depending on the data](https://github.com/kubernetes/kubernetes/issues/106428).
+```
+I1025 00:15:15.525108       1 example.go:116] "Example" data="This is text with a line break\nand \"quotation marks\"." someInt=1 someFloat=0.1 someStruct={StringField: First line,
+second line.}
+```
 
 ### JSON log format
 
@@ -82,7 +139,7 @@ Example of JSON log format (pretty printed):
 
 Keys with special meaning:
 * `ts` - timestamp as Unix time (required, float)
-* `v` - verbosity (required, int, default 0)
+* `v` - verbosity (only for info and not for error messages, int)
 * `err` - error string (optional, string)
 * `msg` - message (required, string)
 
@@ -139,4 +196,5 @@ The `logrotate` tool rotates logs daily, or once the log size is greater than 10
 
 * Read about the [Kubernetes Logging Architecture](/docs/concepts/cluster-administration/logging/)
 * Read about [Structured Logging](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1602-structured-logging)
+* Read about [deprecation of klog flags](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
 * Read about the [Conventions for logging severity](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md)