From 0956df46b023d3dcf8a68fe022ecb93b62a174d1 Mon Sep 17 00:00:00 2001 From: Mark Rossetti Date: Mon, 17 Oct 2022 10:09:17 -0700 Subject: [PATCH 1/3] Updates for WindowsHostProcessContainers going to stable Signed-off-by: Mark Rossetti --- .../feature-gates.md | 14 ++- .../create-hostprocess-pod.md | 94 ++++++++++++++----- 2 files changed, 83 insertions(+), 25 deletions(-) diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 495bb4dcca7d7..ac4ebcdee96cd 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -201,8 +201,6 @@ For a reference to old feature gates that are removed, please refer to | `WinDSR` | `false` | Alpha | 1.14 | | | `WinOverlay` | `false` | Alpha | 1.14 | 1.19 | | `WinOverlay` | `true` | Beta | 1.20 | | -| `WindowsHostProcessContainers` | `false` | Alpha | 1.22 | 1.22 | -| `WindowsHostProcessContainers` | `true` | Beta | 1.23 | | {{< /table >}} ### Feature gates for graduated or deprecated features @@ -324,6 +322,18 @@ For a reference to old feature gates that are removed, please refer to | `WatchBookmark` | `false` | Alpha | 1.15 | 1.15 | | `WatchBookmark` | `true` | Beta | 1.16 | 1.16 | | `WatchBookmark` | `true` | GA | 1.17 | - | +| `WindowsEndpointSliceProxying` | `false` | Alpha | 1.19 | 1.20 | +| `WindowsEndpointSliceProxying` | `true` | Beta | 1.21 | 1.21 | +| `WindowsEndpointSliceProxying` | `true` | GA | 1.22| - | +| `WindowsGMSA` | `false` | Alpha | 1.14 | 1.15 | +| `WindowsGMSA` | `true` | Beta | 1.16 | 1.17 | +| `WindowsGMSA` | `true` | GA | 1.18 | - | +| `WindowsHostProcessContainers` | `false` | Alpha | 1.22 | 1.22 | +| `WindowsHostProcessContainers` | `true` | Beta | 1.23 | 1.25 | +| `WindowsHostProcessContainers` | `true` | GA | 1.26 | - | +| `WindowsRunAsUserName` | `false` | Alpha | 1.16 | 1.16 | +| `WindowsRunAsUserName` | `true` | Beta | 1.17 | 1.17 | +| `WindowsRunAsUserName` | `true` | GA | 1.18 | - | {{< /table >}} ## Using a feature diff --git a/content/en/docs/tasks/configure-pod-container/create-hostprocess-pod.md b/content/en/docs/tasks/configure-pod-container/create-hostprocess-pod.md index ed5506c45b689..dcff3b98c0a45 100644 --- a/content/en/docs/tasks/configure-pod-container/create-hostprocess-pod.md +++ b/content/en/docs/tasks/configure-pod-container/create-hostprocess-pod.md @@ -7,7 +7,7 @@ min-kubernetes-server-version: 1.23 -{{< feature-state for_k8s_version="v1.23" state="beta" >}} +{{< feature-state for_k8s_version="v1.26" state="stable" >}} Windows HostProcess containers enable you to run containerized workloads on a Windows host. These containers operate as @@ -42,7 +42,6 @@ HostProcess containers have access to the host's network interfaces and IP addre - Consolidation of administrative tasks and security policies. This reduces the degree of privileges needed by Windows nodes. - ## {{% heading "prerequisites" %}} @@ -56,24 +55,13 @@ communicate with containerd directly by passing the hostprocess flag via CRI. Yo latest version of containerd (v1.6+) to run HostProcess containers. [How to install containerd.](/docs/setup/production-environment/container-runtimes/#containerd) -To *disable* HostProcess containers you need to pass the following feature gate flag to the -**kubelet** and **kube-apiserver**: - -```powershell ---feature-gates=WindowsHostProcessContainers=false -``` - -See [Features Gates](/docs/reference/command-line-tools-reference/feature-gates/#overview) -documentation for more details. - - - ## Limitations These limitations are relevant for Kubernetes v{{< skew currentVersion >}}: - HostProcess containers require containerd 1.6 or higher - {{< glossary_tooltip text="container runtime" term_id="container-runtime" >}}. + {{< glossary_tooltip text="container runtime" term_id="container-runtime" >}} and + containerd 1.7 is recommended. - HostProcess pods can only contain HostProcess containers. This is a current limitation of the Windows OS; non-privileged Windows containers cannot share a vNIC with the host IP namespace. - HostProcess containers run as a process on the host and do not have any degree of @@ -121,9 +109,7 @@ the configurations which need to be set to enable the creation of a HostProcess hostNetwork -

Will be in host network by default initially. Support - to set network to a different compartment may be desirable in - the future.

+

Pods container HostProcess containers must use the host's network namespace.

Allowed Values

@@ -179,18 +166,33 @@ spec: ## Volume mounts HostProcess containers support the ability to mount volumes within the container volume space. +Volume mount behavior differs depending on the version of containerd runtime used by on the node. + +### Containerd v1.6 + Applications running inside the container can access volume mounts directly via relative or absolute paths. An environment variable `$CONTAINER_SANDBOX_MOUNT_POINT` is set upon container creation and provides the absolute host path to the container volume. Relative paths are based upon the `.spec.containers.volumeMounts.mountPath` configuration. -### Example {#volume-mount-example} +To access service account tokens (for example) the following path structures are supported within the container: + +- `.\var\run\secrets\kubernetes.io\serviceaccount\` +- `$CONTAINER_SANDBOX_MOUNT_POINT\var\run\secrets\kubernetes.io\serviceaccount\` + +### Containerd v1.7 (and greater) -To access service account tokens the following path structures are supported within the container: +Applications running inside the container can access volume mounts directly via the volumeMount's +specified `mountPath` (just like Linux and non-HostProcess Windows containers). -`.\var\run\secrets\kubernetes.io\serviceaccount\` +For backwards compatibility volumes can also be accessed via using the same relative paths configured +by containerd v1.6. -`$CONTAINER_SANDBOX_MOUNT_POINT\var\run\secrets\kubernetes.io\serviceaccount\` +To access service account tokens (for example) use the following paths within the container: + +- `c:\var\run\secrets\kubernetes.io\serviceaccount` +- `/var/run/secrets/kubernetes.io/serviceaccount/` +- `$CONTAINER_SANDBOX_MOUNT_POINT\var\run\secrets\kubernetes.io\serviceaccount\` ## Resource limits @@ -203,6 +205,8 @@ used for resource tracking due to the difference in how HostProcess containers a ## Choosing a user account +### System accounts + HostProcess containers support the ability to run as one of three supported Windows service accounts: - **[LocalSystem](https://docs.microsoft.com/windows/win32/services/localsystem-account)** @@ -215,6 +219,50 @@ malicious) damage to the host. The LocalSystem service account has the highest l of privilege of the three and should be used only if absolutely necessary. Where possible, use the LocalService service account as it is the least privileged of the three options. +### Local accounts {#local-accounts} + +HostProcess containers can also run as local user accounts which allows for node operators to give +fine-grained access to workloads. + +To run HostProcess containers as a local user; A local usergroup must first be created on the node +and the name of that local usergroup must be specified in the `runAsUserName` field in the deployment. +This will cause an new ephemeral local user account to be created, joined to the specified usergroup, +and used by the container. This provides a number a benefits including eliminating the need to manage +passwords for local user accounts. + +{{< note >}} +Running HostProcess containers as local user accounts requires containerd v1.7+ +{{< /note >}} + +Example: + +1. Create a local user group on the node (this can be done in another HostProcess container). + + ```cmd + net localgroup hpc-localgroup /add + ``` + +1. Grant access to desired resources on the node to the local usergroup. + This can be done with tools like [icacls](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/icacls). + +1. Set `runSsUserName` to the name of the local usergroup for the pod or individual containers. + + ```yaml + securityContext: + windowsOptions: + hostProcess: true + runAsUserName: hpc-localgroup + ``` + +1. Schedule the pod! + +## Base Image for HostProcess Containers + +HostProcess containers can be built from any of the existing [Windows Container base images](https://learn.microsoft.com/virtualization/windowscontainers/manage-containers/container-base-images). + +Additionally a new base mage has been created just for HostProcess containers! +For more information please check out the [windows-host-process-containers-base-image github project](https://github.com/microsoft/windows-host-process-containers-base-image#overview). + ## Troubleshooting HostProcess containers - HostProcess containers fail to start with `failed to create user process token: failed to logon user: Access is denied.: unknown` From 31f4963c8d6f2923609ae428cb9c9a3f78b7409a Mon Sep 17 00:00:00 2001 From: Mark Rossetti Date: Tue, 15 Nov 2022 11:21:58 -0800 Subject: [PATCH 2/3] fixing feature-gates.md --- .../command-line-tools-reference/feature-gates.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index ac4ebcdee96cd..7e63d194e42bd 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -322,18 +322,9 @@ For a reference to old feature gates that are removed, please refer to | `WatchBookmark` | `false` | Alpha | 1.15 | 1.15 | | `WatchBookmark` | `true` | Beta | 1.16 | 1.16 | | `WatchBookmark` | `true` | GA | 1.17 | - | -| `WindowsEndpointSliceProxying` | `false` | Alpha | 1.19 | 1.20 | -| `WindowsEndpointSliceProxying` | `true` | Beta | 1.21 | 1.21 | -| `WindowsEndpointSliceProxying` | `true` | GA | 1.22| - | -| `WindowsGMSA` | `false` | Alpha | 1.14 | 1.15 | -| `WindowsGMSA` | `true` | Beta | 1.16 | 1.17 | -| `WindowsGMSA` | `true` | GA | 1.18 | - | | `WindowsHostProcessContainers` | `false` | Alpha | 1.22 | 1.22 | | `WindowsHostProcessContainers` | `true` | Beta | 1.23 | 1.25 | | `WindowsHostProcessContainers` | `true` | GA | 1.26 | - | -| `WindowsRunAsUserName` | `false` | Alpha | 1.16 | 1.16 | -| `WindowsRunAsUserName` | `true` | Beta | 1.17 | 1.17 | -| `WindowsRunAsUserName` | `true` | GA | 1.18 | - | {{< /table >}} ## Using a feature From 1d74e4c3a250d41fb62a760b55bd46a5e4fe5aae Mon Sep 17 00:00:00 2001 From: Mark Rossetti Date: Tue, 15 Nov 2022 11:26:21 -0800 Subject: [PATCH 3/3] Apply suggestions from code review Co-authored-by: Brandon Smith Co-authored-by: Qiming Teng --- .../create-hostprocess-pod.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/content/en/docs/tasks/configure-pod-container/create-hostprocess-pod.md b/content/en/docs/tasks/configure-pod-container/create-hostprocess-pod.md index dcff3b98c0a45..cad26cf29a374 100644 --- a/content/en/docs/tasks/configure-pod-container/create-hostprocess-pod.md +++ b/content/en/docs/tasks/configure-pod-container/create-hostprocess-pod.md @@ -188,7 +188,7 @@ specified `mountPath` (just like Linux and non-HostProcess Windows containers). For backwards compatibility volumes can also be accessed via using the same relative paths configured by containerd v1.6. -To access service account tokens (for example) use the following paths within the container: +As an example, to access service account tokens within the container you would use one of the following paths: - `c:\var\run\secrets\kubernetes.io\serviceaccount` - `/var/run/secrets/kubernetes.io/serviceaccount/` @@ -207,7 +207,7 @@ used for resource tracking due to the difference in how HostProcess containers a ### System accounts -HostProcess containers support the ability to run as one of three supported Windows service accounts: +By default, HostProcess containers support the ability to run as one of three supported Windows service accounts: - **[LocalSystem](https://docs.microsoft.com/windows/win32/services/localsystem-account)** - **[LocalService](https://docs.microsoft.com/windows/win32/services/localservice-account)** @@ -221,14 +221,15 @@ use the LocalService service account as it is the least privileged of the three ### Local accounts {#local-accounts} -HostProcess containers can also run as local user accounts which allows for node operators to give +If configured, HostProcess containers can also run as local user accounts which allows for node operators to give fine-grained access to workloads. To run HostProcess containers as a local user; A local usergroup must first be created on the node and the name of that local usergroup must be specified in the `runAsUserName` field in the deployment. -This will cause an new ephemeral local user account to be created, joined to the specified usergroup, -and used by the container. This provides a number a benefits including eliminating the need to manage -passwords for local user accounts. +Prior to initializing the HostProcess container, a new **ephemeral** local user account to be created and joined to the specified usergroup, from which the container is run. +This provides a number a benefits including eliminating the need to manage passwords for local user accounts. +passwords for local user accounts. An initial HostProcess container running as a service account can be used to +prepare the user groups for later HostProcess containers. {{< note >}} Running HostProcess containers as local user accounts requires containerd v1.7+ @@ -243,9 +244,9 @@ Example: ``` 1. Grant access to desired resources on the node to the local usergroup. - This can be done with tools like [icacls](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/icacls). + This can be done with tools like [icacls](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/icacls). -1. Set `runSsUserName` to the name of the local usergroup for the pod or individual containers. +1. Set `runAsUserName` to the name of the local usergroup for the pod or individual containers. ```yaml securityContext: