From bc7c1dbd9697d3aa0af5e298ed8bc9e174c7d925 Mon Sep 17 00:00:00 2001 From: lakshmi Date: Fri, 6 Oct 2023 16:31:14 +0530 Subject: [PATCH 1/2] added a new concept page Sidecar containers and revise details details of init containers and pod lifecycle. --- .../workloads/pods/init-containers.md | 86 +++++------- .../concepts/workloads/pods/pod-lifecycle.md | 23 +++- .../workloads/pods/sidecar-containers.md | 123 ++++++++++++++++++ 3 files changed, 177 insertions(+), 55 deletions(-) create mode 100644 content/en/docs/concepts/workloads/pods/sidecar-containers.md diff --git a/content/en/docs/concepts/workloads/pods/init-containers.md b/content/en/docs/concepts/workloads/pods/init-containers.md index 2533c286d7907..e9e1b748abfb7 100644 --- a/content/en/docs/concepts/workloads/pods/init-containers.md +++ b/content/en/docs/concepts/workloads/pods/init-containers.md @@ -14,6 +14,9 @@ Init containers can contain utilities or setup scripts not present in an app ima You can specify init containers in the Pod specification alongside the `containers` array (which describes app containers). +In Kubernetes, a [sidecar container](/docs/concepts/workloads/pods/sidecar-containers/) is a container that +starts before the main application container and _continues to run_. This document is about init containers: +containers that run to completion during Pod initialization. @@ -48,14 +51,33 @@ including resource limits, [volumes](/docs/concepts/storage/volumes/), and secur resource requests and limits for an init container are handled differently, as documented in [Resource sharing within containers](#resource-sharing-within-containers). -Also, init containers do not support `lifecycle`, `livenessProbe`, `readinessProbe`, or -`startupProbe` because they must run to completion before the Pod can be ready. +Regular init containers (in other words: excluding sidecar containers) do not support the +`lifecycle`, `livenessProbe`, `readinessProbe`, or `startupProbe` fields. Init containers +must run to completion before the Pod can be ready; sidecar containers continue running +during a Pod's lifetime, and _do_ support some probes. See [sidecar container](/docs/concepts/workloads/pods/sidecar-containers/) +for further details about sidecar containers. If you specify multiple init containers for a Pod, kubelet runs each init container sequentially. Each init container must succeed before the next can run. When all of the init containers have run to completion, kubelet initializes the application containers for the Pod and runs them as usual. +### Differences from sidecar containers + +Init containers run and complete their tasks before the main application container starts. +Unlike [sidecar containers](/docs/concepts/workloads/pods/sidecar-containers), +init containers are not continuously running alongside the main containers. + +Init containers run to completion sequentially, and the main container does not start +until all the init containers have successfully completed. + +init containers do not support `lifecycle`, `livenessProbe`, `readinessProbe`, or +`startupProbe` whereas sidecar containers support all these [probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe) to control their lifecycle. + +Init containers share the same resources (CPU, memory, network) with the main application +containers but do not interact directly with them. They can, however, use shared volumes +for data exchange. + ## Using init containers Because init containers have separate images from app containers, they @@ -289,51 +311,9 @@ The Pod which is already running correctly would be killed by `activeDeadlineSec The name of each app and init container in a Pod must be unique; a validation error is thrown for any container sharing a name with another. -#### API for sidecar containers - -{{< feature-state for_k8s_version="v1.28" state="alpha" >}} - -Starting with Kubernetes 1.28 in alpha, a feature gate named `SidecarContainers` -allows you to specify a `restartPolicy` for init containers which is independent of -the Pod and other init containers. Container [probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe) -can also be added to control their lifecycle. - -If an init container is created with its `restartPolicy` set to `Always`, it will -start and remain running during the entire life of the Pod, which is useful for -running supporting services separated from the main application containers. - -If a `readinessProbe` is specified for this init container, its result will be used -to determine the `ready` state of the Pod. - -Since these containers are defined as init containers, they benefit from the same -ordering and sequential guarantees as other init containers, allowing them to -be mixed with other init containers into complex Pod initialization flows. - -Compared to regular init containers, sidecar-style init containers continue to -run and the next init container can begin starting once the kubelet has set -the `started` container status for the sidecar-style init container to true. -That status either becomes true because there is a process running in the -container and no startup probe defined, or -as a result of its `startupProbe` succeeding. - -This feature can be used to implement the sidecar container pattern in a more -robust way, as the kubelet always restarts a sidecar container if it fails. - -Here's an example of a Deployment with two containers, one of which is a sidecar: - -{{% code_sample language="yaml" file="application/deployment-sidecar.yaml" %}} - -This feature is also useful for running Jobs with sidecars, as the sidecar -container will not prevent the Job from completing after the main container -has finished. - -Here's an example of a Job with two containers, one of which is a sidecar: - -{{% code_sample language="yaml" file="application/job/job-sidecar.yaml" %}} - -#### Resource sharing within containers +### Resource sharing within containers -Given the ordering and execution for init containers, the following rules +Given the order of execution for init, sidecar and app containers, the following rules for resource usage apply: * The highest of any particular resource request or limit defined on all init @@ -354,6 +334,10 @@ limit. Pod level control groups (cgroups) are based on the effective Pod request and limit, the same as the scheduler. +{{< comment >}} +This section also present under [sidecar containers](/docs/concepts/workloads/pods/sidecar-containers/) page. +If you're editing this section, change both places. +{{< /comment >}} ### Pod restart reasons @@ -373,7 +357,9 @@ Kubernetes, consult the documentation for the version you are using. ## {{% heading "whatsnext" %}} -* Read about [creating a Pod that has an init container](/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container) -* Learn how to [debug init containers](/docs/tasks/debug/debug-application/debug-init-containers/) -* Read about an overview of [kubelet](/docs/reference/command-line-tools-reference/kubelet/) and [kubectl](/docs/reference/kubectl/) -* Learn about the [types of probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe): liveness, readiness, startup probe. +Learn more about the following: +* [Creating a Pod that has an init container](/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container). +* [Debug init containers](/docs/tasks/debug/debug-application/debug-init-containers/). +* Overview of [kubelet](/docs/reference/command-line-tools-reference/kubelet/) and [kubectl](/docs/reference/kubectl/). +* [Types of probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe): liveness, readiness, startup probe. +* [Sidecar containers](/docs/concepts/workloads/pods/sidecar-containers). \ No newline at end of file diff --git a/content/en/docs/concepts/workloads/pods/pod-lifecycle.md b/content/en/docs/concepts/workloads/pods/pod-lifecycle.md index 1f73ccbe3ff99..6453bfcb23353 100644 --- a/content/en/docs/concepts/workloads/pods/pod-lifecycle.md +++ b/content/en/docs/concepts/workloads/pods/pod-lifecycle.md @@ -150,11 +150,22 @@ the `Terminated` state. The `spec` of a Pod has a `restartPolicy` field with possible values Always, OnFailure, and Never. The default value is Always. -The `restartPolicy` applies to all containers in the Pod. `restartPolicy` only -refers to restarts of the containers by the kubelet on the same node. After containers -in a Pod exit, the kubelet restarts them with an exponential back-off delay (10s, 20s, -40s, …), that is capped at five minutes. Once a container has executed for 10 minutes -without any problems, the kubelet resets the restart backoff timer for that container. +The `restartPolicy` for a Pod applies to {{< glossary_tooltip text="app containers" term_id="app-container" >}} +in the Pod and to regular [init containers](/docs/concepts/workloads/pods/init-containers/). +[Sidecar containers](/docs/concepts/workloads/pods/sidecar-containers/) +ignore the Pod-level `restartPolicy` field: in Kubernetes, a sidecar is defined as an +entry inside `initContainers` that has its container-level `restartPolicy` set to `Always`. +For init containers that exit with an error, the kubelet restarts the init container if +the Pod level `restartPolicy` is either `OnFailure` or `Always`. + +When the kubelet is handling container restarts according to the configured restart +policy, that only applies to restarts that make replacement containers inside the +same Pod and running on the same node. After containers in a Pod exit, the kubelet +restarts them with an exponential back-off delay (10s, 20s,40s, …), that is capped at +five minutes. Once a container has executed for 10 minutes without any problems, the +kubelet resets the restart backoff timer for that container. +[Sidecar containers and Pod lifecycle](/docs/concepts/workloads/pods/sidecar-containers/#sidecar-containers-and-pod-lifecycle) +explains the behaviour of `init containers` when specify `restartpolicy` field on it. ## Pod conditions @@ -582,6 +593,8 @@ for more details. * Learn more about [container lifecycle hooks](/docs/concepts/containers/container-lifecycle-hooks/). +* Learn more about [sidecar containers](/docs/concepts/workloads/pods/sidecar-containers/). + * For detailed information about Pod and container status in the API, see the API reference documentation covering [`status`](/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodStatus) for Pod. diff --git a/content/en/docs/concepts/workloads/pods/sidecar-containers.md b/content/en/docs/concepts/workloads/pods/sidecar-containers.md new file mode 100644 index 0000000000000..6d36283add31b --- /dev/null +++ b/content/en/docs/concepts/workloads/pods/sidecar-containers.md @@ -0,0 +1,123 @@ +--- +title: Sidecar Containers +content_type: concept +weight: 50 +--- + + +{{< feature-state for_k8s_version="v1.28" state="alpha" >}} + +Sidecar containers are the secondary containers that run along with the main +application container within the same {{< glossary_tooltip text="Pod" term_id="pod" >}}. +These containers are used to enhance or to extend the functionality of the main application +container by providing additional services, or functionality such as logging, monitoring, +security, or data synchronization, without directly altering the primary application code. + + + +## Enabling sidecar containers + +Starting with Kubernetes 1.28, a +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) named +`SidecarContainers` allows you to specify a `restartPolicy` for containers listed in a +Pod's `initContainers` field. These restartable _sidecar_ containers are independent with +other [init containers](/docs/concepts/workloads/pods/init-containers/) and main +application container within the same pod. These can be started, stopped, or restarted +without effecting the main application container and other init containers. + +## Sidecar containers and Pod lifecycle + +If an init container is created with its `restartPolicy` set to `Always`, it will +start and remain running during the entire life of the Pod. This can be helpful for +running supporting services separated from the main application containers. + +If a `readinessProbe` is specified for this init container, its result will be used +to determine the `ready` state of the Pod. + +Since these containers are defined as init containers, they benefit from the same +ordering and sequential guarantees as other init containers, allowing them to +be mixed with other init containers into complex Pod initialization flows. + +Compared to regular init containers, sidecars defined within `initContainers` continue to +run after they have started. This is important when there is more than one entry inside +`.spec.initContainers` for a Pod. After a sidecar-style init container is running (the kubelet +has set the `started` status for that init container to true), the kubelet then starts the +next init container from the ordered `.spec.initContainers` list. +That status either becomes true because there is a process running in the +container and no startup probe defined, or as a result of its `startupProbe` succeeding. + +Here's an example of a Deployment with two containers, one of which is a sidecar: + +{{% code_sample language="yaml" file="application/deployment-sidecar.yaml" %}} + +This feature is also useful for running Jobs with sidecars, as the sidecar +container will not prevent the Job from completing after the main container +has finished. + +Here's an example of a Job with two containers, one of which is a sidecar: + +{{% code_sample language="yaml" file="application/job/job-sidecar.yaml" %}} +By default, this feature is not available in Kubernetes. To avail this feature, you +need to enable the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +named `SidecarContainers`. + +## Differences from regular containers + +Sidecar containers run alongside regular containers in the same pod. However, they do not +execute the primary application logic; instead, they provide supporting functionality to +the main application. + +Sidecar containers have their own independent lifecycles. They can be started, stopped, +and restarted independently of regular containers. This means you can update, scale, or +maintain sidecar containers without affecting the primary application. + +Sidecar containers share the same network and storage namespaces with the primary +container This co-location allows them to interact closely and share resources. + +## Differences from init containers + +Sidecar containers work alongside the main container, extending its functionality and +providing additional services. + +Sidecar containers run concurrently with the main application container. They are active +throughout the lifecycle of the pod and can be started and stopped independently of the +main container. Unlike [init containers](/docs/concepts/workloads/pods/init-containers/), +sidecar containers support [probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe) to control their lifecycle. + +These containers can interact directly with the main application containers, sharing +the same network namespace, filesystem, and environment variables. They work closely +together to provide additional functionality. + +## Resource sharing within containers + +{{< comment >}} +This section is also present in the [init containers](/docs/concepts/workloads/pods/init-containers/) page. +If you're editing this section, change both places. +{{< /comment >}} + +Given the order of execution for init, sidecar and app containers, the following rules +for resource usage apply: + +* The highest of any particular resource request or limit defined on all init + containers is the *effective init request/limit*. If any resource has no + resource limit specified this is considered as the highest limit. +* The Pod's *effective request/limit* for a resource is the higher of: + * the sum of all app containers request/limit for a resource + * the effective init request/limit for a resource +* Scheduling is done based on effective requests/limits, which means + init containers can reserve resources for initialization that are not used + during the life of the Pod. +* The QoS (quality of service) tier of the Pod's *effective QoS tier* is the + QoS tier for init containers and app containers alike. + +Quota and limits are applied based on the effective Pod request and +limit. + +Pod level control groups (cgroups) are based on the effective Pod request and +limit, the same as the scheduler. + +## {{% heading "whatsnext" %}} + +* Read a blog post on [native sidecar containers](/blog/2023/08/25/native-sidecar-containers/). +* Read about [creating a Pod that has an init container](/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container). +* Learn about the [types of probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe): liveness, readiness, startup probe. \ No newline at end of file From b23c499c2398110207649a7276e73ce3e70b59a3 Mon Sep 17 00:00:00 2001 From: lakshmi Date: Mon, 11 Dec 2023 17:33:21 +0530 Subject: [PATCH 2/2] Revise details for Resource sharing within containers --- .../docs/concepts/workloads/pods/init-containers.md | 2 +- .../concepts/workloads/pods/sidecar-containers.md | 11 +++++++---- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/content/en/docs/concepts/workloads/pods/init-containers.md b/content/en/docs/concepts/workloads/pods/init-containers.md index e9e1b748abfb7..480d4cee80b9e 100644 --- a/content/en/docs/concepts/workloads/pods/init-containers.md +++ b/content/en/docs/concepts/workloads/pods/init-containers.md @@ -362,4 +362,4 @@ Learn more about the following: * [Debug init containers](/docs/tasks/debug/debug-application/debug-init-containers/). * Overview of [kubelet](/docs/reference/command-line-tools-reference/kubelet/) and [kubectl](/docs/reference/kubectl/). * [Types of probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe): liveness, readiness, startup probe. -* [Sidecar containers](/docs/concepts/workloads/pods/sidecar-containers). \ No newline at end of file +* [Sidecar containers](/docs/concepts/workloads/pods/sidecar-containers). diff --git a/content/en/docs/concepts/workloads/pods/sidecar-containers.md b/content/en/docs/concepts/workloads/pods/sidecar-containers.md index 6d36283add31b..a1f5a9fbb8fc9 100644 --- a/content/en/docs/concepts/workloads/pods/sidecar-containers.md +++ b/content/en/docs/concepts/workloads/pods/sidecar-containers.md @@ -101,14 +101,16 @@ for resource usage apply: * The highest of any particular resource request or limit defined on all init containers is the *effective init request/limit*. If any resource has no resource limit specified this is considered as the highest limit. -* The Pod's *effective request/limit* for a resource is the higher of: - * the sum of all app containers request/limit for a resource +* The Pod's *effective request/limit* for a resource is the sum of +[pod overhead](/docs/concepts/scheduling-eviction/pod-overhead/) and the higher of: + * the sum of all non-init containers(app and sidecar containers) request/limit for a + resource * the effective init request/limit for a resource * Scheduling is done based on effective requests/limits, which means init containers can reserve resources for initialization that are not used during the life of the Pod. * The QoS (quality of service) tier of the Pod's *effective QoS tier* is the - QoS tier for init containers and app containers alike. + QoS tier for all init, sidecar and app containers alike. Quota and limits are applied based on the effective Pod request and limit. @@ -120,4 +122,5 @@ limit, the same as the scheduler. * Read a blog post on [native sidecar containers](/blog/2023/08/25/native-sidecar-containers/). * Read about [creating a Pod that has an init container](/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container). -* Learn about the [types of probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe): liveness, readiness, startup probe. \ No newline at end of file +* Learn about the [types of probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe): liveness, readiness, startup probe. +* Learn about [pod overhead](/docs/concepts/scheduling-eviction/pod-overhead/).