Questions about possibility to use the role in production

[pablodav]

I'm looking for something to reuse and help to implement kubernetes for windows.

As we know the limits with flannel, I'm thinking in the possibility to use what you have written for optional deployment with some of the working networkings for prod env.

Possibility with static routes:
https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/configuring-host-gateway-mode

More details: https://kubernetes.io/docs/getting-started-guides/windows/

Do you feel it will be good to work on this direction?
Can I use your base and start sending pull requests once have some progress?
Do you have any recommendation before going to modify it?

I'm trying some other projects to like kubespray and kismatic, but both doesn't have any playbook for windows, so what you have written here is very useful to go into a new direction.

Probably this new project could be what I'm was looking for (still not tested, just got into this repository at this time).

[pablodav]

Looks like I missed some parts of flannel, as if it is working as hostgateway, is the same as I have proposed above... right?
So no changes in that direction are needed, only use these ansible playbooks and try to go forward... fix things, etc?

[ptylenda]

@pablodav thanks for your messages - yes, the Ansible playbooks that are in this repository are capable of deploying a proper Flannel pod network, which is also handled by Flanneld on Windows machines. The only difference is that Windows nodes are not hosting Flannel as a pod but as a standalone service.

This can be summed up as:

1. CNI plugins for Windows are required, they have been provided in this pull request, that is NOT merged yet: Windows CNI for overlay (vxlan) and host-gw (l2bridge) modes containernetworking/plugins#85
2. Flannel for Windows with host-gw support is required, this has been provided in this pull request, that is NOT merged yet too: host-gw: add windows support flannel-io/flannel#921
3. I have performed a few small customizations and fixes to these pull requests and they are distributed via github release so that they are available to Ansible playbooks: https://github.com/ptylenda/flannel/releases/download/v0.10.0-win-hostgw-fix/flannel-v0.10.0-windows-amd64.tar.gz https://github.com/ptylenda/plugins/releases/download/v0.7.0-rc2-win-fix/cni-plugins-amd64-v0.7.0-rc2-win-fix.tgz
4. The above have one major issue that I have compiled into them (New-NetNat)[https://docs.microsoft.com/en-us/powershell/module/netnat/new-netnat?view=win10-ps] command in order to experiment with creating outbound NAT forcefully on Hyper-V Virtual Switch. Unfortunately it breaks pod-to-pod communication on Windows :( So it is either having a nice pod-to-pod communication and nodeport services, or having a working NAT for communication to external services. Creating a version without this NAT support would require compling Windows CNI for overlay (vxlan) and host-gw (l2bridge) modes containernetworking/plugins#85 (it is a bit newer than the ones I provide and should have nice fixes for the issues that I spotted) and also Taking first interface address in hostgw instead of the last one - required for kube-proxy in userspace rakelkar/flannel#8 which has my fixes for kube-proxy working in userspace mode.

As to being production ready: the biggest problem with current hybrid Kubernetes clusters is the way how Windows HNS behaves, it is hard to debug and rather unpredictable. So it looks like Kubernetes itself would be ready for production but some underlying Docker/Windows components are unreliable yet.

And of course, you are more than welcome to sending pull requests :) In the next few weeks I have a limited time for introducing changes to this repo, so this is also the reason why I am releasing this code in the current state. I bet that this could be a good starting point for supporting Windows in kubespray later.

For now I can suggest installing Ubuntu 16.04 on a few Hyper-V VMs and Windows Server 1709 on a few Hyper-V VMs. When you have this, enable WinRM on Windows nodes (as in the official guide) so that you can use Ansible. From this point you should be able to use the playbooks, I am not sure if these playbooks will work properly when WITHOUT http proxy, as I was not testing it. But anyway these would be minor fixes.

[pablodav]

Thanks!

This is very complete answer! Better than I was specting.