Support For Windows Containers

[kahootali]

Feature Request

Currently, I know that Minikube is a single node cluster, and that node is Linux so it wont be able to run Windows Containers, but it would be better if Minikube is able to run Windows Containers also by being able to add a Windows node.

[ivanfioravanti]

This would be awesome for testing hybrid platforms.

[ghost]

+1 - This would be an awsome feature so we could test hybrid platforms

[masroorhasan]

+1 would be extremely useful.

[ericmaia]

+1 This would be great.

[dnascimento]

+1 This is critical for create a good developer experience in Windows

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[adamlith]

+1

[kahootali]

/remove-lifecycle stale

[dbwest]

+1. I'd rather not have to work with Windows but it is a fact that it is my corporation's jam. Minikube for Windows would make some ppl's lives better.

[nogna]

+1

[d4nj0nes]

+1

[Amith211]

+1

[david-jarman]

+1

[k-modi]

+1

[n-gauthier-novo]

+1

[kisadam]

+1

[Josefczak]

+1

[goldsam]

+1

[gavinheaden]

+1

[cwchapma]

+1

[amarof]

+1

[aleksandrov]

+1

[ptink]

This would be amazing, there doesn't currently seem to be a good solution out there for local development/testing of multi-node apps

[AmeyK-Globant]

+1, this is highly desirable and would be much helpful.

[ghost]

+1

[Aut0R3V]

@tstromberg I'd like to work on this one. Can you please explain the progress here?

[tstromberg]

As far as I know, no one has attempted to even prototype such a beast. You'd be the first!

If you do decide to embark on this ambitious effort, I suggest doing some prototyping and report on your results on this issue. Once there is a clear path forward, I'd suggest sending out a MEP: https://github.com/kubernetes/minikube/tree/master/enhancements

I'd be very excited to see this happen. /cc @blueelvis

[Aut0R3V]

I went through the stuff here - #116 . I feel like I have some idea on the progress that has been attained until now. Can we form up like a group on slack or something about this? It might be a little hard to pitch in detailed conversations here

[MarkEwer]

I want to up-vote this feature. I have been using Docker Desktop because it is really the only game in town if you need to run a mix of Windows and Linux containers on your developer workstation. But, Docker just upped the price for Docker Desktop to $21 per developer per month. While their product is very strong and I will pay that price to continue developing my product, if MiniKube was an alternative I bet my whole company would switch; especially since we use K8S for our production deployments anyway.

I think Docker's licensing change is an opportunity for MiniKube.

[sharifelgamal]

We don't currently have the bandwidth to implement this feature, but I'm certainly not opposed to supporting Windows containers.

As for people who are looking for alternatives to docker desktop, you can use minikube with a vm driver (like hyperv or virtualbox) and point your local docker CLI to the docker inside of the minikube vm (minikube docker-env).

[vrapolinario]

Getting late to this party, but I think there's definitely interest from the Windows community to see Minikube support Windows nodes. As far as I know Windows nodes require the following on K8s:

• A CNI (could be Flannel) - That's supported on Minikube
• A Windows node - multi-node is supported on Minikube

I'm not a dev, so won't even say I can pick this up, but I'll definitely try to add a separate node via the K8s interface and see how it goes. I'll report back.

[vrapolinario]

Alright, so as expected I hit a wall here.

I have a multi-node MiniKube deployment with flannel with all the pre-requisites for Windows in place. However, the documentation for adding Windows nodes on Kubernetes relies on kubeadm, which is not available on MiniKube: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/

I tried minikube ssh to install kubeadm but I'm neither a Linux or Minukube expert, so that was unsuccessful.
If someone can point me to either how to get kubeadm installed for Minikube or how do I get the join command for Minikube nodes, I can continue this trial and report on that.

[sharifelgamal]

kubeadm is included in every minikube cluster (we use kubeadm to start and administer clusters). It should be in /var/lib/minikube/binaries/v1.23.6/kubeadm.

[vrapolinario]

Thanks @sharifelgamal. This is where my lack of Linux skills show up.
I did find kubeadm, but I'm not able to run it:

PS C:\Users\viniap> minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ cd /var/lib/minikube/binaries/
$ ls
v1.23.3
$ cd v1.23.3/
$ ls
kubeadm  kubectl  kubelet
$ kubeadm token create --print-join-command
-bash: kubeadm: command not found
$

Apologies for such basic ask, but how does one goes about running kubeadm from there?

[vrapolinario]

And just as I wrote this, I found the anwer:

$ ./kubeadm token create --print-join-command
failed to load admin kubeconfig: open /etc/kubernetes/admin.conf: permission denied
To see the stack trace of this error execute with --v=5 or higher
$ sudo ./kubeadm token create --print-join-command
kubeadm join control-plane.minikube.internal:8443 --token omsio0.w7mh9q9z6fxvjtvo --discovery-token-ca-cert-hash sha256:8ceaf34a09e0e5cf4f82d523d3c9068e78d390b221d606dc6f1224400b164f6c
$

Will continue the tests here.

[vrapolinario]

Just wanted to report the progress so far:

PS C:\GitHub\MiniKube Windows Containers> kubectl get node -o wide
NAME           STATUS     ROLES                  AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
minikube       Ready      control-plane,master   66m     v1.23.3   192.168.0.92   <none>        Buildroot 2021.02.4              4.19.202         containerd://1.4.12
minikube-m02   Ready      <none>                 65m     v1.23.3   192.168.0.93   <none>        Buildroot 2021.02.4              4.19.202         containerd://1.4.12
minikube-m03   NotReady   <none>                 6m27s   v1.23.3   192.168.0.95   <none>        Windows Server 2022 Datacenter   10.0.20348.469   containerd://1.6.6

Windows node added successfully, thanks to the help of @jsturtevant and @marosset. I'll now try to configure flannel and kube-proxy, but these are Windows specific tasks. I'll document the whole process once I'm done and publish here.

[vrapolinario]

Another update on this: not only the node is now joined to the cluster, but flannel and kube-proxy are deployed:

PS C:\GitHub\AKSDemo> kubectl get pods -A
NAMESPACE     NAME                                  READY   STATUS              RESTARTS        AGE
kube-system   coredns-64897985d-68c69               1/1     Running             0               173m
kube-system   etcd-minikube                         1/1     Running             0               173m
kube-system   kube-apiserver-minikube               1/1     Running             0               173m
kube-system   kube-controller-manager-minikube      1/1     Running             0               173m
kube-system   kube-flannel-ds-amd64-nqlbj           1/1     Running             0               158m
kube-system   kube-flannel-ds-amd64-rrllx           1/1     Running             0               158m
kube-system   kube-flannel-ds-windows-amd64-7299v   1/1     Running             1 (4m45s ago)   99m
kube-system   kube-proxy-hvx6l                      1/1     Running             0               172m
kube-system   kube-proxy-jt9rx                      1/1     Running             0               173m
kube-system   kube-proxy-windows-8fzps              1/1     Running             1 (4m29s ago)   11m
kube-system   kube-scheduler-minikube               1/1     Running             0               173m
kube-system   storage-provisioner                   1/1     Running             1 (172m ago)    173m

Next I'll try to deploy a simple web workload to see if everything works. I started documenting the steps for all this to work and will share that as well soon.
Once again, thanks @jsturtevant for the help on this!

[vrapolinario]

Another update on this and an ask for help:

• Windows node is up and status is Ready when running kubectl get nodes.
• Windows nodes can run Windows containers. kubectl get pods shows status of Windows containers are Running.

I have documented the steps to get this working here: https://github.com/vrapolinario/MinikubeWindowsContainers

I'm having an issue now when trying to access service deployed for windows container deployment. Both NodePort or LoadBalancer are not working correctly. The Service is deployed correctly, and get an internal IP address. I noticed that when I use either minikube service or minikube tunnel the IP address to which the service goes to is the first Linux node, not the Windows node.

@sharifelgamal or others, any ideas on why this is happening? I'm not sure this is a behavior of Minikube because it is not aware of the Windows node, or if there's an issue in my configuration.

[vrapolinario]

For the sake of this being a proof of concept that actually proves its concept, I have added a section to the documentation showing how to deploy a Windows container and then using kubectl exec to interact with the container. I'd appreciate some help in figuring out the networking piece. Maybe there's a config file in MiniKube on which I can manually add the node or something similar?

[PhilParisot]

That's awesome work @vrapolinario thank you so much for your efforts on this! This feature is desperately needed in order to move away from docker desktop...

[hassanshamshir]

@vrapolinario Have you find any solution so far?

[vrapolinario]

Folks, after a long wait we were finally able to make this work. 😊Thanks @bobsira for the help in solving the networking issue.
https://github.com/vrapolinario/MinikubeWindowsContainers

I'll be updating the Known issues. Also, we're thinking about multiple things:

• We want to make the process of installing the Windows VM more fluid. We should be able to use PowerShell remote for all operations post install.
• The whole process can also be streamlined and we're looking into that.
• In order for the guidance to follow the releases of MiniKube, K8s, Windows, continerd, etc, we should be able to check for these and set them up as variables to ensure compatibility.

The above are things we're going to look into. We'd also love to get some support from the Minikube maintainers to get this incorporated into the official releases. @sharifelgamal thoughts?

[afbjorklund]

@vrapolinario

If you require any big changes to minikube in order to support Windows, I would suggest drafting a "MEP" proposal:

https://github.com/kubernetes/minikube/tree/master/enhancements

Especially the testing part (in CI) might prove something of a challenge.

[afbjorklund]

cc @kubernetes/minikube-maintainers

[vrapolinario]

Thanks @afbjorklund. I think we still have some ways to go before that, but would be nice to get some perspective from the maintainers on this as we move along. Agree that the testing part would be an issue, but something we can look into.

[leeseoungsuk1]

@vrapolinario

When using Windows-related images in Minikube, errors occurred. Therefore, creating a Linux manager node and a Windows worker node might be a solution. The manager node can then allocate tasks to the Windows worker node, preventing errors associated with using Windows images. Is that correct?

[vrapolinario]

In any K8s environment the control plane needs to run on Linux nodes and the worker nodes can be either Linux (for Linux containers) or Windows (for Windows containers). Outside of MiniKube, Windows is supported as a node and Windows containers can be scheduled using either node selector, or taints and toleration.

[leeseoungsuk1]

@vrapolinario

When you apply a .yaml file on a manager node to execute, does Kubernetes distribute the YAML file to worker nodes based on whether it's a Windows or Linux image? Also, I'm curious if you can use both Windows image containers and Linux image containers in the same YAML file.

[mloskot]

After nodeSelector is the simplest way to constrain Pods to nodes with specific labels.

apiVersion: v1
kind: Pod
...
spec:
    nodeSelector:
      kubernetes.io/os: windows

I'm curious if you can use both Windows image containers and Linux image containers in the same YAML file.

Yes, you can have multiple Pod manifests in single YAML file, each manifest can schedule pods to different node, some to Linux node and others to Windows node, as long as container OS compatibility goes. However, this is a common Kubernetes, nothing really specific to Minikube.

[leeseoungsuk1]

Thank you. My curiosity has been satisfied.