Revisit kube-reserved calculation for containerd

[shyamjvs]

Background

kube-reserved is meant to capture resource reservation for kubernetes system daemons (not including pods). For the amazon EKS ami it's mainly just the kubelet and container-runtime today. kube-reserved is typically a function of pod density on the nodes.

The current kube-reserved calculation is 2-3 years old (last updated as part of this experiment we did) when docker (+ shim) was the only container runtime option in the AMI. However, Amazon EKS ended support for Docker starting with the Kubernetes version 1.24 launch. Containerd is the only supported runtime now.

Here's the kube-reserved logic we have in our AMIs today:

amazon-eks-ami/files/bootstrap.sh

Line 214 in 6b1df11

memory_to_reserve=$((11 * $max_num_pods + 255))

amazon-eks-ami/files/bootstrap.sh

Lines 221 to 225 in 6b1df11

	# We are using these CPU ranges from GKE (https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#node_allocatable):
	# 6% of the first core
	# 1% of the next core (up to 2 cores)
	# 0.5% of the next 2 cores (up to 4 cores)
	# 0.25% of any cores above 4 cores

Problem

Per the k8s containerd GA release blog the containerd integration consumes less CPU/memory in comparison with the docker integration (~10-12% - see below). We'll need to repeat the experiment given containerd versions being used in our AMIs are newer (1.5+). But overall we should expect notable savings.

At the steady state, with 105 pods, the containerd 1.1 integration consumes less CPU and memory overall compared to
Docker 18.03 CE integration with dockershim. The results vary with the number of pods running on the node, 105 is 
chosen because it is the current default for the maximum number of user pods per node.

As shown in the figures below, compared to Docker 18.03 CE integration with dockershim, the containerd 1.1
integration has 30.89% lower kubelet cpu usage, 68.13% lower container runtime cpu usage, 11.30% lower kubelet 
resident set size (RSS) memory usage, 12.78% lower container runtime RSS memory usage.

Additional Notes

• With the recent launch of the Prefix Delegation (PD) feature in our AMIs, the supported pod densities have gone up for various instance types. The new logic sets max pod density as 110 for <= 30 vCPU instance-types and 250 for > 30 vCPU instance-types (xref). Discussed more in this issue. While PD should also be taken into account for kube-reserved calculation, that can happen independent from this change IMO since docker -> containerd switch is a common denominator. The drop in baseline usage will help customers reap incremental benefits, even if it means bumping it up later for PD-specific customers
• Another issue is that - the bottlerocket AMI seems to have copied the same kube-reserved calculation as the EKS AMI (xrefs - cpu, mem) despite using containerd. Once we fix the calculation here, we should go have them update it too

[shyamjvs]

@suket22 @mmerkes @natherz97 @jayanthvn @akestner - You've all been close to the ground here. Please add any thoughts you may have.

[bryanasdev000]

Related to #1145

[stevehipwell]

I've just roughly re-run the tests on an AL2 EKS v1.29 node and have the following results.

• Kubelet + containerd use about 65Mb with no pods running
• Each pod with a single container adds around 6Mb to the runtime usage
• 110 pods with 1 container uses about 680MB for the runtime
• 110 pods with 5 containers uses about 1200MB for the runtime