Bottlerocket 1.13.0 moving to cgroup v2 for new variants

[markusboehme]

What is changing?
Starting with Bottlerocket 1.13.0, new variants will default to using the Linux kernel’s cgroup (“control groups”) v2 interface for process organization and enforcing resource limits instead of the cgroup v1 interface used in all prior variants. The Kubernetes 1.26 variants will be the first to default to using cgroup v2:

• aws-k8s-1.26
• aws-k8s-1.26-nvidia
• metal-k8s-1.26
• vmware-k8s-1.26

Previously introduced variants will continue to use cgroup v1 by default until their respective end of life. Hence, the following variants will not observe any change:

• aws-ecs-1
• aws-ecs-1-nvidia
• aws-k8s-1.22
• aws-k8s-1.22-nvidia
• aws-k8s-1.23
• aws-k8s-1.23-nvidia
• aws-k8s-1.24
• aws-k8s-1.24-nvidia
• aws-k8s-1.25
• aws-k8s-1.25-nvidia
• metal-k8s-1.22
• metal-k8s-1.23
• metal-k8s-1.24
• metal-k8s-1.25
• vmware-k8s-1.22
• vmware-k8s-1.23
• vmware-k8s-1.24
• vmware-k8s-1.25

The ecosystem’s support for cgroup v2, also known as “the unified cgroup hierarchy”, has matured over the past years. The change to make cgroup v2 the default for new Bottlerocket variants becomes necessary as projects are heading towards the deprecation of support for cgroup v1.

What does this mean to me?
In general, the move to cgroup v2 is not expected to be disruptive to workloads. This change is an implementation detail relevant to Kubernetes and lower layers mostly. However, containers that directly interact with control groups may require changes to continue working with cgroup v2. Most software needing this direct interaction can be expected to support cgroup v2 if used in sufficiently recent versions.

As one example, systemd interacts with control groups. When running systemd inside a container, you need to ensure the version of systemd inside your container image supports cgroup v2. In this situation, support would depend on general cgroup v2 support of the distribution used for your container’s base image: For example, containers based on Amazon Linux 2 would not be able to run systemd inside them, while containers based on Amazon Linux 2023 would have no such limitation.

This Kubernetes blog post announcing support for cgroup v2 contains further general information on cgroup v2 and tips for migrating workloads.

Help! I found a problem!
If you moved to a variant defaulting to cgroup v2 and suspect a problem related to this change, you can either stay on the new variant and revert back to using cgroup v1, or you can downgrade to a current release of a variant that does not default to cgroup v2 (e.g. aws-k8s-1.25 instead of aws-k8s-1.26).

To disable cgroup v2 and use cgroup v1 instead, you may provide the following settings e.g. via the user data mechanism to instruct systemd on the host to use cgroup v1 and trigger a reboot during first boot to make the change take effect without manual intervention:

[settings.boot]
reboot-to-reconcile = true

[settings.boot.init]
"systemd.unified_cgroup_hierarchy" = ["0"]

If you suspect the problem is with Bottlerocket, please reach out through AWS Premium Support or open a GitHub issue for further guidance.

Can I use cgroup v2 on older variants?
Yes, starting with Bottlerocket 1.13.0 you may opt into using cgroup v2 on e.g. the aws-k8s-1.25 variant even though it defaults to using cgroup v1. All changes required for cgroup v2 are available to all variants. Please note that support for cgroup v2 was only declared generally available starting with Kubernetes 1.25, though.

Similar to opting out of cgroup v2 on variants that default to it, you may opt into cgroup v2 on variants that don't default to it via the following settings e.g. provided via the user data mechanism:

[settings.boot]
reboot-to-reconcile = true

[settings.boot.init]
"systemd.unified_cgroup_hierarchy" = ["1"]