From b701d73113168bb9d4f48a434e513f8db3230eb0 Mon Sep 17 00:00:00 2001 From: Patrick Lang Date: Wed, 27 Mar 2019 01:05:30 +0000 Subject: [PATCH 1/5] Start Windows considerations list for conformance tests --- .../sig-architecture/conformance-tests.md | 46 ++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/contributors/devel/sig-architecture/conformance-tests.md b/contributors/devel/sig-architecture/conformance-tests.md index d726094ed4c..720afa46e05 100644 --- a/contributors/devel/sig-architecture/conformance-tests.md +++ b/contributors/devel/sig-architecture/conformance-tests.md @@ -41,7 +41,7 @@ specifically, a test is eligible for promotion to conformance if: to pre-pull images for conformance tests) - it works without non-standard filesystem permissions granted to pods - it does not rely on any binaries that would not be required for the linux - kernel or kubelet to run (e.g., can't rely on git) + kernel or kubelet to run (e.g., can't rely on git), or differ in behavior based on OS (nslookup, ping) - any container images used within the test support all architectures for which kubernetes releases are built - it passes against the appropriate versions of kubernetes as spelled out in @@ -70,6 +70,50 @@ reasonable production worthy environments: bypass policies that are not yet GA, but which may reasonably be enabled on a conformant cluster (e.g., pod security policy, non-GA scheduler annotations) +### Windows & Linux Considerations + +As of v1.14, Windows is a stable feature, but not yet included in conformance +testing. In preparation for this, a large number of conformance tests are +already included in Windows testing. You can see what tests have already pass by +looking at TestGrid for results of Windows tests running on +[Azure](https://testgrid.k8s.io/sig-windows#aks-engine-azure-windows-master) and +[GCE](https://testgrid.k8s.io/sig-windows#gce-windows-master)). + +Generally speaking, the goals are to: + +- Make sure tests that are already passing remain passing. If new OS-specific +functionality is added, it should be in a new test. +- Ensure that new tests covering Linux-specific functionality are tagged with `[LinuxOnly]` +(see: [Kinds of Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md#kinds-of-tests), + - Give future reviewers a reference to an active issue or documentation clarifying why a test +cannot run on Windows. + +The tests that are running today: + +- Rely only on container images that already have a multi-architecture manifest +including Windows versions, or have been ported by SIG-Windows +(see [kubernetes-sigs/windows-testing/images](https://github.com/kubernetes-sigs/windows-testing/tree/master/images) +- Do not depend on any functionality is different or not available on Windows. The full list +is available in the Windows Kubernetes docs under [api](https://kubernetes.io/docs/setup/windows/intro-windows-in-kubernetes/#api). +A brief summary is included here as a starting point. If the docs are insufficient +or there are more questions, please contact #SIG-Windows on Slack to get another +reviewer. + +Some of the most common differences to watch for are: +- Most user & security context options are not supported since Windows does not +use POSIX-style users, permissions or ACLs +- Privileged containers and host networking modes are not supported. Containers are always isolated. +- Only NTFS is supported. Volume mounts specifying other filesystems (ext4, xfs) or mediums (memory) are not supported + - Projected mounts and permission masks are not supported + - Mappings of individual files are not supported. This means some features that rely on it such as +V1.Container.terminationMessagePath or writing service entries to `/etc/hosts` also do not work. +- Network and DNS settings must be passed through CNI. Windows does not use `/etc/resolv.conf` +and stores networking settings in the Windows registry instead +- Windows treats all DNS lookups with a `.` to be FQDN, not PQDN. +- Windows uses job objects or Hyper-V for pod isolation and resource controls, not CGroups. These are managed implicitly by Docker or ContainerD, not by the kubelet. Do not check properties of CGroups as pass/fail criteria. +- ICMP only works between pods on the same network, and are not routable to external networks. TCP/UDP are routable. + + ## Conformance Test Version Skew Policy As each new release of Kubernetes provides new functionality, the subset of From 2d844e6ffacd6ba6431ddcd2ca228bf972e945de Mon Sep 17 00:00:00 2001 From: Patrick Lang Date: Tue, 2 Apr 2019 13:16:17 -0700 Subject: [PATCH 2/5] Merge in Claudiu's list from #3522, review feedback --- .../sig-architecture/conformance-tests.md | 52 ++++++++++++------- 1 file changed, 33 insertions(+), 19 deletions(-) diff --git a/contributors/devel/sig-architecture/conformance-tests.md b/contributors/devel/sig-architecture/conformance-tests.md index 720afa46e05..ed571c0dee8 100644 --- a/contributors/devel/sig-architecture/conformance-tests.md +++ b/contributors/devel/sig-architecture/conformance-tests.md @@ -41,7 +41,8 @@ specifically, a test is eligible for promotion to conformance if: to pre-pull images for conformance tests) - it works without non-standard filesystem permissions granted to pods - it does not rely on any binaries that would not be required for the linux - kernel or kubelet to run (e.g., can't rely on git), or differ in behavior based on OS (nslookup, ping) + kernel or kubelet to run (e.g., can't rely on git) +- it does not depend on outputs that change based on OS (nslookup, ping, chmod, ls) - any container images used within the test support all architectures for which kubernetes releases are built - it passes against the appropriate versions of kubernetes as spelled out in @@ -72,12 +73,15 @@ reasonable production worthy environments: ### Windows & Linux Considerations -As of v1.14, Windows is a stable feature, but not yet included in conformance -testing. In preparation for this, a large number of conformance tests are +As of v1.14, Windows is a stable feature, but not required in all Kubernetes +cluster and is therefore not part of the conformance suite. Nonetheless, it's +importatnt to verify that the behavior of Windows nodes conformance as much as +possible. To that end, a large number of conformance tests are already included in Windows testing. You can see what tests have already pass by -looking at TestGrid for results of Windows tests running on -[Azure](https://testgrid.k8s.io/sig-windows#aks-engine-azure-windows-master) and -[GCE](https://testgrid.k8s.io/sig-windows#gce-windows-master)). +looking at TestGrid for results of Windows tests running on +[Azure](https://testgrid.k8s.io/sig-windows#aks-engine-azure-windows-master) and +[GCE](https://testgrid.k8s.io/sig-windows#gce-windows-master)). Tests may be +scheduled for any PR with the bot command `/test pull-kubernetes-e2e-aks-engine-azure-windows`. Generally speaking, the goals are to: @@ -93,26 +97,36 @@ The tests that are running today: - Rely only on container images that already have a multi-architecture manifest including Windows versions, or have been ported by SIG-Windows (see [kubernetes-sigs/windows-testing/images](https://github.com/kubernetes-sigs/windows-testing/tree/master/images) -- Do not depend on any functionality is different or not available on Windows. The full list +- Do not depend on any functionality that is different or not available on Windows. The full list is available in the Windows Kubernetes docs under [api](https://kubernetes.io/docs/setup/windows/intro-windows-in-kubernetes/#api). A brief summary is included here as a starting point. If the docs are insufficient or there are more questions, please contact #SIG-Windows on Slack to get another reviewer. Some of the most common differences to watch for are: -- Most user & security context options are not supported since Windows does not -use POSIX-style users, permissions or ACLs -- Privileged containers and host networking modes are not supported. Containers are always isolated. -- Only NTFS is supported. Volume mounts specifying other filesystems (ext4, xfs) or mediums (memory) are not supported - - Projected mounts and permission masks are not supported - - Mappings of individual files are not supported. This means some features that rely on it such as -V1.Container.terminationMessagePath or writing service entries to `/etc/hosts` also do not work. -- Network and DNS settings must be passed through CNI. Windows does not use `/etc/resolv.conf` -and stores networking settings in the Windows registry instead -- Windows treats all DNS lookups with a `.` to be FQDN, not PQDN. -- Windows uses job objects or Hyper-V for pod isolation and resource controls, not CGroups. These are managed implicitly by Docker or ContainerD, not by the kubelet. Do not check properties of CGroups as pass/fail criteria. -- ICMP only works between pods on the same network, and are not routable to external networks. TCP/UDP are routable. +- Container Options & Actions + - Pod SecurityContext is set. Most of the fields are Linux specific, and any field set in the Pod's SecurityContext will result in the Pod not being able to spawn or not work as intended. + - Privileged containers are not supported. Containers are always isolated. + - Windows uses job objects or Hyper-V for pod isolation and resource controls, not CGroups. These are managed +implicitly by Docker or ContainerD, not by the kubelet. Do not check properties of CGroups as pass/fail criteria. + - Running Linux-specific commands are not likely to work. Some commands may work using a Windows [busybox](https://github.com/kubernetes-sigs/windows-testing/tree/master/images/busybox) container. +- Storage + - File permissions cannot be set on volumes. Tests using `DefaultMode` or `Mode` and checking the resulting permissions will fail. + - Only NTFS is supported. Volume mounts specifying other filesystems (ext4, xfs) or mediums (memory) are not supported + - Mappings of individual files are not supported. Tests which are mounting or expecting such files to be mounted (including /etc/hosts, /etc/resolv.conf, /dev/termination-log) will fail. + - Bidirectional mount propagation, specifically propagating mounts from a container to host, does not work. +- Networking + - Pods set `HostNetwork=true`. Is not supported on Windows, and the Pod will not start. + - Network and DNS settings must be passed through CNI. Windows does not use `/etc/resolv.conf`, so tests should + not rely on reading that file to check DNS settings. + - Windows treats all DNS lookups with a `.` to be FQDN, not PQDN. For example `kubernetes` will resolve as a PQDN, + but `kubernetes.default` will be resolved as a FQDN and fail. + - ICMP only works between pods on the same network, and are not routable to external networks. TCP/UDP are routable. + - Windows containers do not support IPv6. + +The existing tests which are affected by one of those criteria are tagged with `[LinuxOnly]` +(see: [Kinds of Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md#kinds-of-tests). ## Conformance Test Version Skew Policy From 1bc880849b78a0f51cede64ad392cb622812a917 Mon Sep 17 00:00:00 2001 From: Patrick Lang Date: Tue, 9 Apr 2019 12:30:40 -0700 Subject: [PATCH 3/5] More updates based on review feedback --- .../devel/sig-architecture/conformance-tests.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/contributors/devel/sig-architecture/conformance-tests.md b/contributors/devel/sig-architecture/conformance-tests.md index ed571c0dee8..c949321757a 100644 --- a/contributors/devel/sig-architecture/conformance-tests.md +++ b/contributors/devel/sig-architecture/conformance-tests.md @@ -73,12 +73,9 @@ reasonable production worthy environments: ### Windows & Linux Considerations -As of v1.14, Windows is a stable feature, but not required in all Kubernetes -cluster and is therefore not part of the conformance suite. Nonetheless, it's -importatnt to verify that the behavior of Windows nodes conformance as much as -possible. To that end, a large number of conformance tests are -already included in Windows testing. You can see what tests have already pass by -looking at TestGrid for results of Windows tests running on +Windows node support is an optional but stable feature as of Kubernetes 1.14. This means that it is +not required by conformance testing. Nonetheless, it's important to verify that the behavior of Windows nodes match the behaviors tested in the conformance suite as much as possible. To that end, a +large number of conformance tests are already included in Windows testing. You can see what tests are already passing by looking at TestGrid for results of Windows tests running on [Azure](https://testgrid.k8s.io/sig-windows#aks-engine-azure-windows-master) and [GCE](https://testgrid.k8s.io/sig-windows#gce-windows-master)). Tests may be scheduled for any PR with the bot command `/test pull-kubernetes-e2e-aks-engine-azure-windows`. @@ -89,7 +86,7 @@ Generally speaking, the goals are to: functionality is added, it should be in a new test. - Ensure that new tests covering Linux-specific functionality are tagged with `[LinuxOnly]` (see: [Kinds of Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md#kinds-of-tests), - - Give future reviewers a reference to an active issue or documentation clarifying why a test +- Give future reviewers a reference to an active issue or documentation clarifying why a test cannot run on Windows. The tests that are running today: @@ -105,15 +102,17 @@ reviewer. Some of the most common differences to watch for are: +- Container Images + - Watch out for image names hardcoded into test cases or YAML files. These are often Linux-only. Instead, they should be adding to or using existing images from [tests/utils/image/manifest.go](https://github.com/kubernetes/kubernetes/blob/master/test/utils/image/manifest.go). This allows the container registry to be configured to one containing Windows images, and also supports testing on clusters with no internet access using a private registry. Multi-arch images supporting Windows are also acceptable. - Container Options & Actions - Pod SecurityContext is set. Most of the fields are Linux specific, and any field set in the Pod's SecurityContext will result in the Pod not being able to spawn or not work as intended. - Privileged containers are not supported. Containers are always isolated. - Windows uses job objects or Hyper-V for pod isolation and resource controls, not CGroups. These are managed implicitly by Docker or ContainerD, not by the kubelet. Do not check properties of CGroups as pass/fail criteria. - - Running Linux-specific commands are not likely to work. Some commands may work using a Windows [busybox](https://github.com/kubernetes-sigs/windows-testing/tree/master/images/busybox) container. + - Running Linux-specific commands are not likely to work. Some commands may work using a Windows [busybox](https://github.com/kubernetes-sigs/windows-testing/tree/master/images/busybox) container. The paths of these binaries may differ from Linux, so it's best to rely on `PATH` rather than using Linux-specific paths such as `/usr/bin/nc` - Storage - File permissions cannot be set on volumes. Tests using `DefaultMode` or `Mode` and checking the resulting permissions will fail. - - Only NTFS is supported. Volume mounts specifying other filesystems (ext4, xfs) or mediums (memory) are not supported + - Only NTFS volumes are supported. Volume mounts specifying other filesystems (ext4, xfs) or mediums (memory) are not supported - Mappings of individual files are not supported. Tests which are mounting or expecting such files to be mounted (including /etc/hosts, /etc/resolv.conf, /dev/termination-log) will fail. - Bidirectional mount propagation, specifically propagating mounts from a container to host, does not work. - Networking From 1d114a224053553ac1a9c118d12daea1a8b770cb Mon Sep 17 00:00:00 2001 From: Patrick Lang Date: Thu, 11 Apr 2019 14:28:40 -0700 Subject: [PATCH 4/5] More review feedback --- .../devel/sig-architecture/conformance-tests.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/contributors/devel/sig-architecture/conformance-tests.md b/contributors/devel/sig-architecture/conformance-tests.md index c949321757a..b714deaae86 100644 --- a/contributors/devel/sig-architecture/conformance-tests.md +++ b/contributors/devel/sig-architecture/conformance-tests.md @@ -42,7 +42,7 @@ specifically, a test is eligible for promotion to conformance if: - it works without non-standard filesystem permissions granted to pods - it does not rely on any binaries that would not be required for the linux kernel or kubelet to run (e.g., can't rely on git) -- it does not depend on outputs that change based on OS (nslookup, ping, chmod, ls) +- where possible, it does not depend on outputs that change based on OS (nslookup, ping, chmod, ls) - any container images used within the test support all architectures for which kubernetes releases are built - it passes against the appropriate versions of kubernetes as spelled out in @@ -109,7 +109,7 @@ Some of the most common differences to watch for are: - Privileged containers are not supported. Containers are always isolated. - Windows uses job objects or Hyper-V for pod isolation and resource controls, not CGroups. These are managed implicitly by Docker or ContainerD, not by the kubelet. Do not check properties of CGroups as pass/fail criteria. - - Running Linux-specific commands are not likely to work. Some commands may work using a Windows [busybox](https://github.com/kubernetes-sigs/windows-testing/tree/master/images/busybox) container. The paths of these binaries may differ from Linux, so it's best to rely on `PATH` rather than using Linux-specific paths such as `/usr/bin/nc` + - Running Linux-specific commands are not likely to work. Some commands may work using a Windows [busybox](https://github.com/kubernetes-sigs/windows-testing/tree/master/images/busybox) container. The paths of these binaries may differ from Linux, so it's best to rely on `PATH` rather than using Linux-specific paths such as `/usr/bin/nc`. - Storage - File permissions cannot be set on volumes. Tests using `DefaultMode` or `Mode` and checking the resulting permissions will fail. - Only NTFS volumes are supported. Volume mounts specifying other filesystems (ext4, xfs) or mediums (memory) are not supported @@ -117,8 +117,8 @@ implicitly by Docker or ContainerD, not by the kubelet. Do not check properties - Bidirectional mount propagation, specifically propagating mounts from a container to host, does not work. - Networking - Pods set `HostNetwork=true`. Is not supported on Windows, and the Pod will not start. - - Network and DNS settings must be passed through CNI. Windows does not use `/etc/resolv.conf`, so tests should - not rely on reading that file to check DNS settings. + - Network and DNS settings must be passed through CNI. Windows does not use `/etc/resolv.conf`, so tests should not rely on reading that file to check DNS settings. + - If you to check network settings such as dns search lists, please use [agnhost](https://github.com/kubernetes/kubernetes/tree/master/test/images/agnhost) to output needed data from the container. Agnhost is a simple golang app that will return the same results for both Windows & Linux for each included command. - Windows treats all DNS lookups with a `.` to be FQDN, not PQDN. For example `kubernetes` will resolve as a PQDN, but `kubernetes.default` will be resolved as a FQDN and fail. - ICMP only works between pods on the same network, and are not routable to external networks. TCP/UDP are routable. @@ -211,6 +211,10 @@ To promote a test to the conformance test suite, open a PR as follows: all of the required [conformance test comment metadata] - add the PR to SIG Architecture's [Conformance Test Review board] +Once you create the PR, please schedule the additional Windows tests with +`/test pull-kubernetes-e2e-aks-engine-azure-windows` to see if any existing tests +that pass on Windows are broken by the change. + ### Conformance Test Comment Metadata From 4de900588a4c78671e1720a6e61191a3d7423dba Mon Sep 17 00:00:00 2001 From: Patrick Lang Date: Thu, 11 Apr 2019 15:58:04 -0700 Subject: [PATCH 5/5] Add suggestion to use agnhost for new scenarios --- contributors/devel/sig-architecture/conformance-tests.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contributors/devel/sig-architecture/conformance-tests.md b/contributors/devel/sig-architecture/conformance-tests.md index b714deaae86..3814f164774 100644 --- a/contributors/devel/sig-architecture/conformance-tests.md +++ b/contributors/devel/sig-architecture/conformance-tests.md @@ -109,7 +109,7 @@ Some of the most common differences to watch for are: - Privileged containers are not supported. Containers are always isolated. - Windows uses job objects or Hyper-V for pod isolation and resource controls, not CGroups. These are managed implicitly by Docker or ContainerD, not by the kubelet. Do not check properties of CGroups as pass/fail criteria. - - Running Linux-specific commands are not likely to work. Some commands may work using a Windows [busybox](https://github.com/kubernetes-sigs/windows-testing/tree/master/images/busybox) container. The paths of these binaries may differ from Linux, so it's best to rely on `PATH` rather than using Linux-specific paths such as `/usr/bin/nc`. + - Running Linux-specific commands are not likely to work. Some commands may work using a Windows [busybox](https://github.com/kubernetes-sigs/windows-testing/tree/master/images/busybox) container. The paths of these binaries may differ from Linux, so it's best to rely on `PATH` rather than using Linux-specific paths such as `/usr/bin/nc`. As an alternative, you can use commands in the cross-platform [agnhost](https://github.com/kubernetes/kubernetes/tree/master/test/images/agnhost) image which is designed to return the same results regardless of OS. - Storage - File permissions cannot be set on volumes. Tests using `DefaultMode` or `Mode` and checking the resulting permissions will fail. - Only NTFS volumes are supported. Volume mounts specifying other filesystems (ext4, xfs) or mediums (memory) are not supported @@ -118,7 +118,7 @@ implicitly by Docker or ContainerD, not by the kubelet. Do not check properties - Networking - Pods set `HostNetwork=true`. Is not supported on Windows, and the Pod will not start. - Network and DNS settings must be passed through CNI. Windows does not use `/etc/resolv.conf`, so tests should not rely on reading that file to check DNS settings. - - If you to check network settings such as dns search lists, please use [agnhost](https://github.com/kubernetes/kubernetes/tree/master/test/images/agnhost) to output needed data from the container. Agnhost is a simple golang app that will return the same results for both Windows & Linux for each included command. + - If you to check network settings such as dns search lists, please use [agnhost](https://github.com/kubernetes/kubernetes/tree/master/test/images/agnhost) to output needed data from the container. - Windows treats all DNS lookups with a `.` to be FQDN, not PQDN. For example `kubernetes` will resolve as a PQDN, but `kubernetes.default` will be resolved as a FQDN and fail. - ICMP only works between pods on the same network, and are not routable to external networks. TCP/UDP are routable.