Feature/kubespray k8swin

[pablodav]

This closes #4 refs to #1, #3 #2

First of all, thanks a lot @ptylenda for this project, it helped me a lot and I was very impressed with the quality and amount of work done here.

After many hours fixing some small things and testing the role I have decided to reduce the focus of this role, and to focus only on windows node integration part.

Trying to reuse more parts from external projects to deploy k8s on linux and focusing this project only on the required parts to add a windows node to the existing kubernetes cluster. For that reason I have removed most parts of k8s for linux and started to deploy k8s with kubespray then adding this project as integrated part of that to deploy win_node, but the modifications are done thinking in the possibility to integrate with any other k8s project that uses kubeadm (this project uses kubeadm to generate tokens to join the win_node).

This is the first merge request, probably needs more work but I think its good to start discussing about what I'm trying to do now.

[ptylenda]

@pablodav Thanks a lot! I will try to look through these changes in a few next days (I am sorry that it takes some time, but I have very limited time resources recently :( )

[pablodav]

Thanks for taking time to respond and consider it.

I don't have read much about k8s and networking, until now I was focused on deploying k8s and all the required stuff. So probably will implement more changes once I have all clear.

Now I have just started to read some parts of the networking stuff around k8s.

Based on:
https://chrislovecnm.com/kubernetes/cni/choosing-a-cni-provider/
https://github.com/kubernetes-incubator/kubespray/blob/master/docs/flannel.md
https://github.com/kubernetes-incubator/kubespray/blob/master/docs/calico.md

And more reading in some other places.

Also based on this:

$ cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.233.64.0/18
FLANNEL_SUBNET=10.233.64.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
$ kubectl run test --image=busybox --command -- tail -f /dev/null
deployment.apps "test" created
$ kubectl get pod
NAME                    READY     STATUS    RESTARTS   AGE
test-5c46448499-6m8gz   1/1       Running   0          18s
$ kubectl describe po test-5c46448499-6m8gz | grep ^IP
IP:             10.233.66.13
$ ping 10.233.66.13
PING 10.233.66.13 (10.233.66.13) 56(84) bytes of data.
64 bytes from 10.233.66.13: icmp_seq=1 ttl=63 time=0.397 ms
64 bytes from 10.233.66.13: icmp_seq=2 ttl=63 time=0.351 ms
^C
--- 10.233.66.13 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.351/0.374/0.397/0.023 ms

I see part of flannel is working, but no service ip is there....

$ kubectl get service
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.233.0.1   <none>        443/TCP   4d
$ ip add | grep 10.233.0.1

Nothing diplayed, also service clusterIP doesn't respond.

It's not about kubernetes-for-windows, but I have noticed on previous deploys kubernetes-for-windows started ip in scope 10.233.0.x, probably because you already solved some part of flannel that I don't understand.

Also I see that flannel probably is not the best option at least for on-premises, or for my own case. Looks like calico is better choice for me.

So probably in my environment I will investigate possibility to have option 2 as: calico in masters and linux nodes, and host-to-host manual network for windows nodes?
Do you think it is possible any other combination? like calico on masters and linux and flannel on windows?

I'm not trying to waste your time in providing the information I'm looking for, but I think you probably have better idea in the way to go than me now, I'm just investigating options and trying to put all in this project to provide good place to start for anyone else.

[pablodav]

Just updating my own notes about what I'm understanding better:

Looks like I have missed some part of k8s networking.

Testing with steps in: https://medium.com/google-cloud/understanding-kubernetes-networking-services-f0cb48e4cc82

service works, not reachable from hosts but reachable from pods. (for linux pods, not tested now in windows)

Looks like I need ingress as next step.

[ptylenda]

@pablodav tons of superb work, thanks a lot! I will be checking all the changes soon on a real environment (merging them already though).

As to flannel vs calico - my initial aim was using canal (calico + flannel), but ~6 months ago calico was not ready for windows yet. As far as I remember, there were some official posts announcing support soon but that is all what I have seen.

There is a lot going on when it comes to NAT settings for pods on windows - this was the biggest problem back in the days when I was implementing these playbooks. Your problems with pingning cluster IPs may be a result of CNI plugin / flanneld compiled by me (if you are using this custom binaries of course), where I have hacked some workarounds when trying to make NAT (external communication from windows pods, to be precise) working. So there was a constant tradeoff: either you have NAT and nice external communication, or you have internal communication, but no contact with external world.

[ptylenda]

BTW I see that 2 days ago kubernetes 1.11 was released, which has some good news: if we switch to Docker 17.10+ on windows, we should have better DNS propagation to windows pods.