Support CRI-ContainerD on Windows

[PatrickLang]

Enhancement Description

• One-line enhancement description: Support CRI-ContainerD on Windows
• KEP: 20190424-windows-cri-containerd
• Primary contact (assignee): @marosset
• Responsible SIGs: sig-windows, sig-node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.18
  • Beta release target (x.y): 1.19 or later
  • Stable release target (x.y): 1.20 or later

PRs & Bugs

The most up-to-date list is on this project board https://github.com/orgs/kubernetes/projects/34.

PRs for 1.18 alpha

• Doc PR doc: Containerd alpha support on Windows website#19208
• Add Windows Containerd/Containerd test passes kubernetes#84822 - switch to branch that containerd 1.4 will ship from

Completed PRs/Issues - these span work in 1.17/18

• Check for terminated reason appropriate for containerD and dockershim kubernetes#82878
• Windows: Fixes termination-file mounting support for containerd kubernetes#83057
• tests: Delete Grace Period should be submitted and removed fails on Windows containerd kubernetes#84610 - will be resolved by Replace nginx image with agnhost in sig-node pods tests kubernetes#84788
• Windows CRI-ContainerD: some paths have / instead of \\ kubernetes#85142 - this is blocking dozens of test cases, can't declare alpha until resolved
• CPU requests values are mutually exclusive on Windows. This should be respected when using containerd. kubernetes#84804 - cpu/memory resource validation has changed, needs to be special cased in Windows kubelet (PR Fix Cpu Requests priority Windows. kubernetes#85856)
• tests: Fixes tests for Windows (containerd, RunAsUserName) kubernetes#83058
• Enable CRI plugin for Windows builds as well containerd/containerd#4076

PRs for beta

• Doc PR v1.19 Windows OS version support + containerd doc updates website#21652
• Add GMSA support for V2 process isolated containers microsoft/hcsshim#797 - Add GMSA support for v2 process isolated containers
• Add annotations to CRI ImageSpec objects kubernetes#90061 - Add ability to pass runtimeclass hander to CRI call for Windows (through annotations for now)
• Add GMSA credential spec passing containerd/cri#1505 - GMSA credential spec passing
• Support named pipe mounts for Windows containers containerd/cri#1511 - Named pipe mounts for windows

PRs for GA

• Doc PR(s) - doc updates for graduating Windows + containerd support to stable website#24862
  • Updates to https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/ for using containerd
  • Updates to https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#cri-containerd
  • Updates to https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#feature-restrictions

Bugs for beta (presumably 1.19)

• tests: should be able to pull from private registry with secret on Windows containerd kubernetes#84609 - failing tests pulling from private registry. This passes on Windows Server 2019, currently failing on 1909. Not an alpha blocker
• Add support for named pipe mounts for Windows containerd/containerd#4320 - named pipe mount support in containerd

Bugs for GA

• [WCOW] Support CredentialSpec containerd/cri#1276 - GMSA support is needed for feature parity with Docker
  • Add GMSA support for V2 process isolated containers microsoft/hcsshim#797 - Add GMSA support for V2 process isolated containers
• SMB mount does not work on containerd on Windows kubernetes#94213 - SMB mount does not work on containerd on Windows
  • fixing issue where SMB share paths cannot resolve with CRI-containerD on Windows kubernetes#96396 - SMB fixes for in-tree volume mounts
  • fix: add trailing backslash to remote path when linking to SMB shares kubernetes-csi/csi-proxy#98 - SMB fixes for csi-proxy volume mounts

Bugs requiring Windows OS fixes

• Windows containers do not have entries in the hosts file kubernetes#70189 - single file mounts cannot overwrite files that exist in container

Tests

Based on Windows Server 2019

• Flannel-L2Bridge - https://testgrid.k8s.io/sig-windows#containerd-l2bridge-windows-master
• Flannel-Overlay - https://testgrid.k8s.io/sig-windows#containerd-overlay-windows-master
• GCE - https://testgrid.k8s.io/sig-windows#gce-windows-2019-containerd-master
• Azure
  • https://testgrid.k8s.io/sig-windows#aks-engine-azure-windows-master-containerd
  • https://testgrid.k8s.io/sig-windows#aks-engine-azure-windows-master-containerd-serial-slow

/sig windows
/sig node
/kind feature

[kacole2]

/milestone v1.15
/stage alpha

[justaugustus]

/assign @PatrickLang

[kacole2]

@PatrickLang , Kubernetes 1.15 Enhancement Freeze is 4/30/2019. To be included in the Kubernetes 1.15 milestone, KEPs are required to be merged and in an "Implementable" state with proper test plans and graduation criteria. Please submit any PRs needed to make this KEP adhere to inclusion criteria. If this will slip from the 1.15 milestone, please let us know so we can make appropriate tracking changes.

[makoscafee]

Hey, @PatrickLang 👋 I'm the v1.15 docs Lead.
Does this enhancement require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

[kacole2]

Hi @PatrickLang . Code Freeze is Thursday, May 30th 2019 @ EOD PST. All enhancements going into the release must be code-complete, including tests, and have docs PRs open.

Please list all current k/k PRs so they can be tracked going into freeze. If the PRs aren't merged by freeze, this feature will slip for the 1.15 release cycle. Only release-blocking issues and PRs will be allowed in the milestone.

If you know this will slip, please reply back and let us know. Thanks!

[kacole2]

Hi @PatrickLang , today is code freeze. I do not see a reply for any PRs to track for this merge. It's now being marked as At Risk in the 1.15 Enhancement Tracking Sheet. If there is no response, or you respond with PRs to track and they are not merged by EOD PST, this will be dropped from the 1.15 Milestone. After this point, only release-blocking issues and PRs will be allowed in the milestone with an exception.

[PatrickLang]

/milestone v1.16
Still not ready. PRs and testing are still in progress. Pushing out.

[evillgenius75]

Hi @PatrickLang , I'm a 1.16 Enhancement Shadow. Is this feature going to be released in alpha stage in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[PatrickLang]

/milestone v1.17
Not likely to complete in 1.16, still having issues getting containerd work on Windows moving forward.

[marosset]

/stage stable

[lippertmarkus]

Is there a documentation similar to https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/ on how to set up a node with containerd with a CNI and join it to the cluster? I was able to install containerd and join the node by kubeadm join with --cri-socket "npipe:////./pipe/containerd-containerd" but after adding flannel as CNI I get the following when scheduling a workload:

Warning  FailedCreatePodSandBox  4m1s                 kubelet, minion    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2d458087deb7544c67f6dccbc3702cd9938489380fc48da5e5038f1ea0fc6bdf": error while ProvisionEndpoint(2d458087deb7544c67f6dccbc3702cd9938489380fc48da5e5038f1ea0fc6bdf_flannel.4096,7BB417A4-9BC3-4006-9374-D9EC6DA41D32,2d458087deb7544c67f6dccbc3702cd9938489380fc48da5e5038f1ea0fc6bdf): failed to find HNSEndpoint 2d458087deb7544c67f6dccbc3702cd9938489380fc48da5e5038f1ea0fc6bdf_flannel.4096: Endpoint 2d458087deb7544c67f6dccbc3702cd9938489380fc48da5e5038f1ea0fc6bdf_flannel.4096 not found

When using host-local CNI instead the pods at least start successfully but don't have any network adapters when running ipconfig.

[TBBle]

That doesn't look like a containerd-related issue, but that Flannel was unable to set up the endpoint. Check the logs of the kube-flannel-ds-windows-amd64 pod to see if it gave you any useful HNS-related failure messages.

[jsturtevant]

@lippertmarkus Could you open up a separate issue to discuss? Don't want to pollute the enhancement issue.

[MorrisLaw]

Hey @PatrickLang @marosset 👋 -- 1.20 Enhancements Shadow here. Just a friendly reminder that the Enhancements Freeze deadline is October 6th.

Do we still plan on graduating this to GA in 1.20?

Also, if time permits, can we please update the KEP to match the new format detailed in https://github.com/kubernetes/enhancements/tree/master/keps/NNNN-kep-template?

Thanks for all of your hard work everyone!

Regards,
Jeremy

[marosset]

@MorrisLaw - Yes we plan to go GA in 1.20.
I can update the KEP to match the new format this week.

[MorrisLaw]

Hi @marosset,

Since your Enhancement is scheduled to be in 1.20, please keep in mind the important upcoming dates:
Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Best,
Jeremy

[marosset]

Thanks @MorrisLaw!
I updated the initial issue comment to include PRs/bugfixes needed for GA.

[eagleusb]

Hello @marosset 👋, 1.20 Docs shadow here.

Does this enhancement work planned for 1.20 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.20 branch in the k/website repo.

This PR can be just a placeholder at this time and must be created before Nov 6th

Also take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[eagleusb]

Hi @marosset 👋

The docs placeholder deadline is almost here.

Please make sure to create a placeholder PR against the dev-1.20 branch in the k/website before the deadline.

Also, please keep in mind the important upcoming dates:

• Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
• Thursday, Nov 12th: Week 9 - Code Freeze
• Monday, Nov 23rd: Week 11 - Docs PR Ready for Review

As a reminder, please link all of your k/k PR as well as docs PR to this issue for the release team to track.

Thanks !

[marosset]

Hi @eagleusb Here is the doc PR for graduating to stable
kubernetes/website#24862

[MorrisLaw]

Hey @marosset 👋

Is this PR the last PR remaining to get this to GA in 1.20 or are there some other PRs that we should explicitly link to this issue?

Please keep in mind that we have these upcoming dates:

• Thursday, Nov 12th: Week 9 - Code Freeze
• Monday, Nov 23rd: Week 11 - Docs PR Ready for Review
  

If we miss the the code freeze, an exception will need to be made if we want to include it in the 1.20 milestone.

If there are some relevant k/k PRs, please link it here so we can accurately track them. Thank you Mark!

[marosset]

@MorrisLaw I linked one more PR for 1.20 (kubernetes/kubernetes#96396)
This is approved and I'm finishing up a little bit of extra manual validation around this right now.

[kikisdeliveryservice]

Hi @marosset

It looks like all the PRs are finished. How does kubernetes/kubernetes#70189 fit in?

Are you expecting to fix this as well? Or fix at a later date?

Thanks!
Kirsten

[marosset]

Hi @kikisdeliveryservice We will need to address that at a later date as fixing this requires an Windows OS fix that is not yet released yet. This functionality also doesn't work with any other container runtimes on Windows so we are not treating it as a blocker from declaring containerD support on Windows stable.

The OS fixes are expected to be available in Nov or Dec 2020 so we'll target fixing kubernetes/kubernetes#70189 in v1.21

@michmike - FYI

[kikisdeliveryservice]

Great thanks for the update!

[marosset]

/close

[k8s-ci-robot]

@marosset: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.