Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new concept page for Sidecar containers #43346

Merged
merged 2 commits into from
Dec 11, 2023
Merged

Add a new concept page for Sidecar containers #43346

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bc7c1db
added a new concept page Sidecar containers and revise details detail…
T-Lakshmi Oct 6, 2023
b23c499
Revise details for Resource sharing within containers
T-Lakshmi Dec 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 36 additions & 50 deletions content/en/docs/concepts/workloads/pods/init-containers.md
View file
Open in desktop
sftim marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ Init containers can contain utilities or setup scripts not present in an app ima
You can specify init containers in the Pod specification alongside the `containers`
array (which describes app containers).

sftim marked this conversation as resolved.
Show resolved Hide resolved
In Kubernetes, a [sidecar container](/docs/concepts/workloads/pods/sidecar-containers/) is a container that
starts before the main application container and _continues to run_. This document is about init containers:
containers that run to completion during Pod initialization.

<!-- body -->

Expand Down Expand Up @@ -48,14 +51,33 @@ including resource limits, [volumes](/docs/concepts/storage/volumes/), and secur
resource requests and limits for an init container are handled differently,
as documented in [Resource sharing within containers](#resource-sharing-within-containers).

Also, init containers do not support `lifecycle`, `livenessProbe`, `readinessProbe`, or
`startupProbe` because they must run to completion before the Pod can be ready.
Regular init containers (in other words: excluding sidecar containers) do not support the
`lifecycle`, `livenessProbe`, `readinessProbe`, or `startupProbe` fields. Init containers
must run to completion before the Pod can be ready; sidecar containers continue running
during a Pod's lifetime, and _do_ support some probes. See [sidecar container](/docs/concepts/workloads/pods/sidecar-containers/)
for further details about sidecar containers.

If you specify multiple init containers for a Pod, kubelet runs each init
container sequentially. Each init container must succeed before the next can run.
When all of the init containers have run to completion, kubelet initializes
the application containers for the Pod and runs them as usual.

### Differences from sidecar containers

Init containers run and complete their tasks before the main application container starts.
Unlike [sidecar containers](/docs/concepts/workloads/pods/sidecar-containers),
init containers are not continuously running alongside the main containers.

Init containers run to completion sequentially, and the main container does not start
until all the init containers have successfully completed.

init containers do not support `lifecycle`, `livenessProbe`, `readinessProbe`, or
`startupProbe` whereas sidecar containers support all these [probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe) to control their lifecycle.

Init containers share the same resources (CPU, memory, network) with the main application
containers but do not interact directly with them. They can, however, use shared volumes
for data exchange.

## Using init containers

Because init containers have separate images from app containers, they
Expand Down Expand Up @@ -289,51 +311,9 @@ The Pod which is already running correctly would be killed by `activeDeadlineSec
The name of each app and init container in a Pod must be unique; a
validation error is thrown for any container sharing a name with another.

#### API for sidecar containers

{{< feature-state for_k8s_version="v1.28" state="alpha" >}}

Starting with Kubernetes 1.28 in alpha, a feature gate named `SidecarContainers`
allows you to specify a `restartPolicy` for init containers which is independent of
the Pod and other init containers. Container [probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe)
can also be added to control their lifecycle.

If an init container is created with its `restartPolicy` set to `Always`, it will
start and remain running during the entire life of the Pod, which is useful for
running supporting services separated from the main application containers.

If a `readinessProbe` is specified for this init container, its result will be used
to determine the `ready` state of the Pod.

Since these containers are defined as init containers, they benefit from the same
ordering and sequential guarantees as other init containers, allowing them to
be mixed with other init containers into complex Pod initialization flows.

Compared to regular init containers, sidecar-style init containers continue to
run and the next init container can begin starting once the kubelet has set
the `started` container status for the sidecar-style init container to true.
That status either becomes true because there is a process running in the
container and no startup probe defined, or
as a result of its `startupProbe` succeeding.

This feature can be used to implement the sidecar container pattern in a more
robust way, as the kubelet always restarts a sidecar container if it fails.

Here's an example of a Deployment with two containers, one of which is a sidecar:

{{% code_sample language="yaml" file="application/deployment-sidecar.yaml" %}}

This feature is also useful for running Jobs with sidecars, as the sidecar
container will not prevent the Job from completing after the main container
has finished.

Here's an example of a Job with two containers, one of which is a sidecar:

{{% code_sample language="yaml" file="application/job/job-sidecar.yaml" %}}

#### Resource sharing within containers
### Resource sharing within containers

Given the ordering and execution for init containers, the following rules
Given the order of execution for init, sidecar and app containers, the following rules
for resource usage apply:

* The highest of any particular resource request or limit defined on all init
Expand All @@ -354,6 +334,10 @@ limit.
Pod level control groups (cgroups) are based on the effective Pod request and
limit, the same as the scheduler.

{{< comment >}}
This section also present under [sidecar containers](/docs/concepts/workloads/pods/sidecar-containers/) page.
If you're editing this section, change both places.
{{< /comment >}}

### Pod restart reasons

Expand All @@ -373,7 +357,9 @@ Kubernetes, consult the documentation for the version you are using.

## {{% heading "whatsnext" %}}

* Read about [creating a Pod that has an init container](/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container)
* Learn how to [debug init containers](/docs/tasks/debug/debug-application/debug-init-containers/)
* Read about an overview of [kubelet](/docs/reference/command-line-tools-reference/kubelet/) and [kubectl](/docs/reference/kubectl/)
* Learn about the [types of probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe): liveness, readiness, startup probe.
Learn more about the following:
* [Creating a Pod that has an init container](/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container).
* [Debug init containers](/docs/tasks/debug/debug-application/debug-init-containers/).
* Overview of [kubelet](/docs/reference/command-line-tools-reference/kubelet/) and [kubectl](/docs/reference/kubectl/).
* [Types of probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe): liveness, readiness, startup probe.
* [Sidecar containers](/docs/concepts/workloads/pods/sidecar-containers).
23 changes: 18 additions & 5 deletions content/en/docs/concepts/workloads/pods/pod-lifecycle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,11 +150,22 @@ the `Terminated` state.
The `spec` of a Pod has a `restartPolicy` field with possible values Always, OnFailure,
and Never. The default value is Always.

The `restartPolicy` applies to all containers in the Pod. `restartPolicy` only
refers to restarts of the containers by the kubelet on the same node. After containers
in a Pod exit, the kubelet restarts them with an exponential back-off delay (10s, 20s,
40s, …), that is capped at five minutes. Once a container has executed for 10 minutes
without any problems, the kubelet resets the restart backoff timer for that container.
The `restartPolicy` for a Pod applies to {{< glossary_tooltip text="app containers" term_id="app-container" >}}
in the Pod and to regular [init containers](/docs/concepts/workloads/pods/init-containers/).
[Sidecar containers](/docs/concepts/workloads/pods/sidecar-containers/)
ignore the Pod-level `restartPolicy` field: in Kubernetes, a sidecar is defined as an
entry inside `initContainers` that has its container-level `restartPolicy` set to `Always`.
For init containers that exit with an error, the kubelet restarts the init container if
the Pod level `restartPolicy` is either `OnFailure` or `Always`.

When the kubelet is handling container restarts according to the configured restart
policy, that only applies to restarts that make replacement containers inside the
same Pod and running on the same node. After containers in a Pod exit, the kubelet
restarts them with an exponential back-off delay (10s, 20s,40s, …), that is capped at
five minutes. Once a container has executed for 10 minutes without any problems, the
kubelet resets the restart backoff timer for that container.
[Sidecar containers and Pod lifecycle](/docs/concepts/workloads/pods/sidecar-containers/#sidecar-containers-and-pod-lifecycle)
explains the behaviour of `init containers` when specify `restartpolicy` field on it.

## Pod conditions

Expand Down Expand Up @@ -582,6 +593,8 @@ for more details.

* Learn more about [container lifecycle hooks](/docs/concepts/containers/container-lifecycle-hooks/).

* Learn more about [sidecar containers](/docs/concepts/workloads/pods/sidecar-containers/).

* For detailed information about Pod and container status in the API, see
the API reference documentation covering
[`status`](/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodStatus) for Pod.
126 changes: 126 additions & 0 deletions content/en/docs/concepts/workloads/pods/sidecar-containers.md
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either in this PR or a follow up, let's change the diagram in https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-lifetime to show the containers in a Pod using something like a Gantt chart.

Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
---
title: Sidecar Containers
content_type: concept
weight: 50
T-Lakshmi marked this conversation as resolved.
Show resolved Hide resolved
---

<!-- overview -->
{{< feature-state for_k8s_version="v1.28" state="alpha" >}}
sftim marked this conversation as resolved.
Show resolved Hide resolved

Sidecar containers are the secondary containers that run along with the main
application container within the same {{< glossary_tooltip text="Pod" term_id="pod" >}}.
These containers are used to enhance or to extend the functionality of the main application
container by providing additional services, or functionality such as logging, monitoring,
security, or data synchronization, without directly altering the primary application code.

<!-- body -->

T-Lakshmi marked this conversation as resolved.
Show resolved Hide resolved
## Enabling sidecar containers

Starting with Kubernetes 1.28, a
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) named
`SidecarContainers` allows you to specify a `restartPolicy` for containers listed in a
Pod's `initContainers` field. These restartable _sidecar_ containers are independent with
other [init containers](/docs/concepts/workloads/pods/init-containers/) and main
application container within the same pod. These can be started, stopped, or restarted
without effecting the main application container and other init containers.

## Sidecar containers and Pod lifecycle

T-Lakshmi marked this conversation as resolved.
Show resolved Hide resolved
If an init container is created with its `restartPolicy` set to `Always`, it will
start and remain running during the entire life of the Pod. This can be helpful for
running supporting services separated from the main application containers.
T-Lakshmi marked this conversation as resolved.
Show resolved Hide resolved

If a `readinessProbe` is specified for this init container, its result will be used
to determine the `ready` state of the Pod.
T-Lakshmi marked this conversation as resolved.
Show resolved Hide resolved

Since these containers are defined as init containers, they benefit from the same
ordering and sequential guarantees as other init containers, allowing them to
be mixed with other init containers into complex Pod initialization flows.

Compared to regular init containers, sidecars defined within `initContainers` continue to
run after they have started. This is important when there is more than one entry inside
`.spec.initContainers` for a Pod. After a sidecar-style init container is running (the kubelet
has set the `started` status for that init container to true), the kubelet then starts the
next init container from the ordered `.spec.initContainers` list.
That status either becomes true because there is a process running in the
container and no startup probe defined, or as a result of its `startupProbe` succeeding.

Here's an example of a Deployment with two containers, one of which is a sidecar:

{{% code_sample language="yaml" file="application/deployment-sidecar.yaml" %}}

This feature is also useful for running Jobs with sidecars, as the sidecar
container will not prevent the Job from completing after the main container
has finished.

Here's an example of a Job with two containers, one of which is a sidecar:

{{% code_sample language="yaml" file="application/job/job-sidecar.yaml" %}}
By default, this feature is not available in Kubernetes. To avail this feature, you
need to enable the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
named `SidecarContainers`.

## Differences from regular containers

Sidecar containers run alongside regular containers in the same pod. However, they do not
execute the primary application logic; instead, they provide supporting functionality to
the main application.

Sidecar containers have their own independent lifecycles. They can be started, stopped,
and restarted independently of regular containers. This means you can update, scale, or
maintain sidecar containers without affecting the primary application.

Sidecar containers share the same network and storage namespaces with the primary
container This co-location allows them to interact closely and share resources.

## Differences from init containers

Sidecar containers work alongside the main container, extending its functionality and
providing additional services.

Sidecar containers run concurrently with the main application container. They are active
throughout the lifecycle of the pod and can be started and stopped independently of the
main container. Unlike [init containers](/docs/concepts/workloads/pods/init-containers/),
sidecar containers support [probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe) to control their lifecycle.

These containers can interact directly with the main application containers, sharing
the same network namespace, filesystem, and environment variables. They work closely
together to provide additional functionality.

sftim marked this conversation as resolved.
Show resolved Hide resolved
## Resource sharing within containers

T-Lakshmi marked this conversation as resolved.
Show resolved Hide resolved
{{< comment >}}
This section is also present in the [init containers](/docs/concepts/workloads/pods/init-containers/) page.
If you're editing this section, change both places.
{{< /comment >}}

Given the order of execution for init, sidecar and app containers, the following rules
for resource usage apply:

* The highest of any particular resource request or limit defined on all init
containers is the *effective init request/limit*. If any resource has no
resource limit specified this is considered as the highest limit.
* The Pod's *effective request/limit* for a resource is the sum of
[pod overhead](/docs/concepts/scheduling-eviction/pod-overhead/) and the higher of:
* the sum of all non-init containers(app and sidecar containers) request/limit for a
resource
* the effective init request/limit for a resource
* Scheduling is done based on effective requests/limits, which means
init containers can reserve resources for initialization that are not used
during the life of the Pod.
* The QoS (quality of service) tier of the Pod's *effective QoS tier* is the
QoS tier for all init, sidecar and app containers alike.

Quota and limits are applied based on the effective Pod request and
limit.

Pod level control groups (cgroups) are based on the effective Pod request and
limit, the same as the scheduler.

## {{% heading "whatsnext" %}}

* Read a blog post on [native sidecar containers](/blog/2023/08/25/native-sidecar-containers/).
* Read about [creating a Pod that has an init container](/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container).
* Learn about the [types of probes](/docs/concepts/workloads/pods/pod-lifecycle/#types-of-probe): liveness, readiness, startup probe.
* Learn about [pod overhead](/docs/concepts/scheduling-eviction/pod-overhead/).