From b4a07b2894e7207e8deada9b14a5ffee8230fddb Mon Sep 17 00:00:00 2001 From: Francesco Romani Date: Mon, 17 Jul 2023 07:42:22 +0200 Subject: [PATCH] WIP: blog: node: kubelet podresources API GA in 1.28 I'm intentionally covering multiple related enhancements with a single blog post. Enhancements: - https://github.com/kubernetes/enhancements/issues/606 - https://github.com/kubernetes/enhancements/issues/2403 - https://github.com/kubernetes/enhancements/issues/3743 Signed-off-by: Francesco Romani --- .../2023-MM-DD-kubelet-podresources-api-ga.md | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 content/en/blog/_posts/2023-MM-DD-kubelet-podresources-api-ga.md diff --git a/content/en/blog/_posts/2023-MM-DD-kubelet-podresources-api-ga.md b/content/en/blog/_posts/2023-MM-DD-kubelet-podresources-api-ga.md new file mode 100644 index 0000000000000..1e7feccbc4b30 --- /dev/null +++ b/content/en/blog/_posts/2023-MM-DD-kubelet-podresources-api-ga.md @@ -0,0 +1,61 @@ +--- +layout: blog +title: 'Kubernetes v1.28: Kubelet podresources API GA' +date: 2023-MM-DD +slug: kubelet-podresources-api-GA +--- + +**Author:** +Francesco Romani (Red Hat) + +The podresources API is an API served by the kubelet locally on the node, which exposes the compute resources exclusively +allocated to containers. In Kubernetes 1.28 the API is now Generally Available. + +## What problem does it solve? + +The kubelet can allocate exclusive resources to containers, like +[CPUs, granting exclusive access to full cores](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3570-cpumanager) +or [memory, either regions or hugepages](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1769-memory-manager). +Workloads which require high performance, or low latency (or both) leverage these features. +The kubelet also can assign [devices to containers](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3573-device-plugin) +Collectively, these features which enable exclusive assignments are known as "resource managers". + +The podresources API was [initially proposed to enable device monitoring](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/606-compute-device-assignment#motivation). +In order to enable monitoring agents, a key prerequisite is to enable introspection of device assignment, which is performed by the kubelet. +Serving this purpose was the initial goal of the API. The API was initially extremely simple, with just a single function implemented, `List`, +to return information about the assignment of devices to containers. + +Without an API like podresources, the only possible option to learn about resource assignment was to read the state files the +resource managers use. While done out of necessity, the problem with this approach is the path and the format of these file are +both internal implementation details. Albeit very stable, the project reserves the chance to change them freely. +Consuming the content of the state files is thus fragile and unsupported, and projects doing this are recommended to consider +moving to podresources API or to other supported APIs. + +Since its inception, the podresources API increased its scope to cover other resource managers than device manager. +In Kubernetes 1.20, the `List` API reports also CPU cores and memory regions (including hugepages); the API also +reports the NUMA locality of the devices, while the locality of CPUs and memory can be inferred from the system. +In kubernetes 1.21, [the API gained the `GetAllocatableResources` function](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2403-pod-resources-allocatable-resources/README.md). +This new API complements the existing `List` API and enables monitoring agents to determine the unallocated resources, +thus enabling new features built on top of the podresources API like a +[NUMA-aware scheduler plugin](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/noderesourcetopology/README.md). + +## Consuming the API + +TBD + +## Future enhancements + +For historical reasons, the podresources API has a less precise specification with respect to the other kubernetes API. +This leads to unspecified behavior in corner cases. +[An effort is ongoing](https://github.com/kubernetes/kubernetes/issues/119423) to rectify this state and to have a more precise specification. + +The [Dynamic Resource Allocation - DRA infrastructure](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3063-dynamic-resource-allocation) +is a major overhaul of the resource management. +[Integration with the podresources API](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3695-pod-resources-for-dra) +is already ongoing. + +## Getting involved + +This feature is driven by the [SIG Node](https://github.com/Kubernetes/community/blob/master/sig-node/README.md) community. +Please join us to connect with the community and share your ideas and feedback around the above feature and +beyond. We look forward to hearing from you!