Merge pull request #4 from pablodav/feature/kubespray_k8win1803

Feature/kubespray k8win1803

README.md

@@ -39,11 +39,11 @@ The original *kubernetes-for-windows* was modified by @pablodav trying to reuse
     - [ ] Communication with external IPs (i.e. outbound NAT) from Windows pods - **this is the most significant issue, with current Windows HNS and Hyper-V Virtual Switch it is not possible to achieve outbound NAT without losing pod-to-pod communication from Windows nodes**. 
  3. There are problems with automatic configuration of DNS in Windows pods (depends on Windows version). Some workarounds have been posted in this [azure-acs-engine issue](https://github.com/Azure/acs-engine/issues/2027).
  4. It is not possible to use Ansible Remote provisioner with Ansible 2.5.0 and Packer 1.2.2 for Windows nodes due to the following exception:
-```
+```shell
 ntlm: HTTPSConnectionPool(host='127.0.0.1', port=63008): Max retries exceeded with url: /wsman (Caused by SSLError(SSLError(1, u'[SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:590)'),))
 ```
 Similar issues are present with Ubuntu templates:
-```
+```shell
 SSH Error: data could not be sent to remote host \"127.0.0.1\". Make sure this host can be reached over ssh
 ```
 Unfortunately I did not have time to investigate this issue yet, but the Packer provisioning process used to work on lower versions of Ansible and Packer.
@@ -206,7 +206,7 @@ Now you have all the roles and variables ready, just install the requirements fr
 sudo pip install -r roles/3d/kubespray/requirements.txt
 ```
 
-### Step 3 - Install Kubernetes packages using roles.kubernetes.yml playbook
+### Step 4 - Install Kubernetes packages using roles.kubernetes.yml playbook
 
 ---
 
@@ -228,6 +228,17 @@ And then the windows nodes:
 ansible-playbook roles.kubernetes.yml -i inventory/kubernetes.ini --tags role::kubernetes-for-windows -b -vvv
 ```
 
+You will notice that the role with tag `role::kubernetes-for-windows` will patch kube-proxy and kube-flannel:
+
+```shell
+kubectl get ds -n kube-system
+NAME           DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
+kube-flannel   3         3         3         3            3           beta.kubernetes.io/os=linux   34m
+kube-proxy     3         3         3         3            3           beta.kubernetes.io/os=linux   8d
+```
+
+So the daemonset will not be deployed in windows.
+
 The playbook consists of the following stages:
 
 1. Installation of common modules on Windows and Linux. This includes various packages required by Kubernetes and recommended configuration, setting up proxy variables, updating OS, changing hostname. (For linux it installs from kubespray modules)
@@ -333,3 +344,43 @@ Then handle these variables appropriately in playbook, set environment variables
 [linux]
 127.0.0.1 ansible_connection=local
 ```
+
+Additional notes about windows performance
+==========================================
+
+Exclude windows defender on docker path and exe files:
+
+```shell
+Add-MpPreference -ExclusionPath C:\ProgramData\docker\
+set-MpPreference -ExclusionProcess "dockerd.exe, flanneld.exe, kube-proxy.exe, kubelet.exe"
+```
+
+Or exclude the docker path in your antivirus.
+
+In case of doubt with windows defender, disable it temporarly:
+
+```shell
+Set-MpPreference -DisableRealtimeMonitoring $true
+```
+
+Some other troubleshooting tweaks
+=================================
+
+Networking with kube-proxy and flanneld services
+------------------------------------------------
+
+Service start order matters, in some tests I (pablodav) have confirmed that this order is required to get all network devices and IP addresses created during start:
+
+1. docker
+2. kubelet
+3. kube-proxy
+4. flanneld
+
+For that reason I have added serialized dependencies on nssm service config on tasks.
+
+Publish services on Premises
+============================
+
+Use this model for on-premises:
+
+https://medium.com/@kyralak/accessing-kubernetes-services-without-ingress-nodeport-or-loadbalancer-de6061b42d72

ansible/inventory/group_vars/k8s-cluster-local/k8s-win.yml

@@ -6,7 +6,8 @@ kubernetes_cluster_domain: "{{ cluster_name }}"  # same as kubespray cluster_nam
 
 kubernetes_reset_master: False
 
-# Choose network plugin ( calico, flannel)
+# Choose network plugin ( calico, flannel, ovn)
+# For now only flannel works in kubernetes-for-windows
 # Can also be set to 'cloud', which lets the cloud provider setup appropriate routing
 # kube_network_plugin: flannel
 
@@ -28,7 +29,7 @@ win_choco_proxy_username: ""
 win_choco_proxy_password: ""
 
 # Set True when you want common/ubuntu to disable swap if preset
-# Or disable swap manually
+# Or disable swap manually (You must disable swap)
 k8s_linux_disable_swap: False
 
 # Use this var if you are going to enable Hype-V isolated containers features

ansible/roles/windows/docker/defaults/main.yml

@@ -19,4 +19,4 @@ win_docker_version: "17.06.2-ee-13"  # "17.06.2-ee-6" "17.06.2-ee-8" "17.10.0-ee
 win_docker_zip_filename: "docker-17-06-2-ee-13.zip"
 win_docker_temp_dir: "/Users/{{ ansible_user }}/Appdata/Local/Temp/DockerMsftProvider"
 win_docker_zip_file: "{{ win_docker_temp_dir }}/{{ win_docker_zip_filename }}"
-win_docker_pull_images: True
+win_docker_pull_images: True  # Image core will be pulled when building windows/pause image anyway

ansible/roles/windows/flannel/defaults/main.yml

@@ -1,5 +1,6 @@
 ---
 flannel_package_url: "https://github.com/ptylenda/flannel/releases/download/v0.10.0-win-hostgw-fix/flannel-v0.10.0-windows-amd64.tar.gz"
+flannel_package_file: "flannel-v0.10.0-windows-amd64.tar.gz"
 flannel_installation_path: "C:/k/"
 flannel_config_path: "C:/etc/kube-flannel/"  # This is the recommended directory - flanneld will automatically load configuration
 flannel_log_path: "C:/var/log/"
@@ -9,4 +10,6 @@ download_temp_path: "C:/Temp/"
 unzip_temp_path: "C:/Temp/flannel/"
 
 kubernetes_pod_network_cidr: "10.244.0.0/16"
-kubernetes_config_path: "C:/etc/kubernetes/kubelet.conf"
+kubernetes_config_path: "C:/etc/kubernetes/kubelet.conf"
+
+kube_proxy_service_name: kube-proxy # used as dependency on flanneld service

ansible/roles/windows/flannel/tasks/main.yml

@@ -14,11 +14,17 @@
     recurse: yes
   tags: cni
 
+- name: check if cni file is downloaded
+  win_stat:
+    path: "{{ download_temp_path }}/{{ flannel_package_file }}"
+  register: flannel_stat_package_file_info
+
 - name: Download flannel package with host-gw support
   win_get_url:
     url: "{{ flannel_package_url }}"
-    dest: "{{ download_temp_path }}/flannel.tar.gz"
+    dest: "{{ download_temp_path }}/{{ flannel_package_file }}"
     force: yes
+  when: not flannel_stat_package_file_info.stat.exists
   tags: cni
 
 - name: Ensure that unzip temporary directory exists
@@ -31,7 +37,7 @@
 # Unzipping tar.gz on windows host is rather clumsy. Another option is to use 7zip directly as in: https://stackoverflow.com/questions/1359793/programmatically-extract-tar-gz-in-a-single-step-on-windows-with-7zip
 #- name: Unzip flannel gz package 
 #  win_unzip:
-#    src: "{{ download_temp_path }}/flannel.tar.gz"
+#    src: "{{ download_temp_path }}/{{ flannel_package_file }}"
 #    dest: "{{ download_temp_path }}"
 #    delete_archive: yes
 #
@@ -49,15 +55,10 @@
 #  with_items: "{{ flannel_find_state.files }}"
 
 - name: Unzip flannel package
-  win_shell: 7z x "{{ download_temp_path }}/flannel.tar.gz" -so | 7z x -aoa -si -ttar -o"{{ unzip_temp_path }}"
+  win_shell: 7z x "{{ download_temp_path }}/{{ flannel_package_file }}" -so | 7z x -aoa -si -ttar -o"{{ unzip_temp_path }}"
   args:
     executable: cmd  # Required as powershell provides unwanted buffering
-  tags: cni
-
-- name: Delete flannel package archive
-  win_file:
-    path: "{{ download_temp_path }}/flannel.tar.gz"
-    state: absent
+  when: not flannel_stat_package_file_info.stat.exists
   tags: cni
 
 - name: Ensure that flannel installation directory exists
@@ -72,6 +73,7 @@
     remote_src: yes
     src: "{{ unzip_temp_path }}"
     dest: "{{ flannel_installation_path }}"
+  when: not flannel_stat_package_file_info.stat.exists
   tags: cni
 
 - name: Delete temporary unzip directory
@@ -123,7 +125,7 @@
         stderr_file: "{{ flannel_log_path }}/{{ flannel_service_name }}-stderr.log"
         state: present
         start_mode: auto
-        #dependencies: kubelet,docker  # latest NSSM throws a lot of errors when used with ansible and multiple dependencies. Defining them manually below
+        dependencies: "{{ kube_proxy_service_name }}"  # latest NSSM throws a lot of errors when used with ansible and multiple dependencies. Defining them manually below
 
     - name: Set flannel service restart delay for short executions
       win_shell: "nssm set {{ flannel_service_name }} AppThrottle 1500"
@@ -132,17 +134,19 @@
       win_shell: "nssm set {{ flannel_service_name }} AppExit Default Restart"
 
     - name: Set flannel service restart delay in case of failure
-      win_shell: "nssm set {{ flannel_service_name }} AppRestartDelay 1000"
+      win_shell: "nssm set {{ flannel_service_name }} AppRestartDelay 5000"
 
     - name: Set flannel service to not overwrite stdout file
       win_shell: "nssm set {{ flannel_service_name }} AppStdoutCreationDisposition 4"
 
     - name: Set flannel service to not overwrite stderr file
       win_shell: "nssm set {{ flannel_service_name }} AppStderrCreationDisposition 4"
 
-    - name: Set flannel service dependency on docker
-      win_shell: "nssm set {{ flannel_service_name }} DependOnService :Docker"
+    #- name: Set flannel service dependency on docker
+    #  win_shell: "nssm set {{ flannel_service_name }} DependOnService :Docker"
 
-    - name: Set flannel service dependency on kubelet
-      win_shell: "nssm set {{ flannel_service_name }} DependOnService +kubelet"
+    #- name: Set flannel service dependency on kubelet
+    #  win_shell: "nssm set {{ flannel_service_name }} DependOnService +kubelet"
+    #- name: Set flannel service dependency on kubelet
+    #  win_shell: "nssm set {{ flannel_service_name }} DependOnService +kube-proxy"      
   tags: cni

ansible/roles/windows/kubernetes-node/defaults/main.yml

@@ -10,7 +10,7 @@ kubelet_log_path: "C:/var/log/"
 kubelet_service_name: kubelet
 
 kube_proxy_log_path: "C:/var/log/"
-kube_proxy_service_name: kube-proxy
+kube_proxy_service_name: kube-proxy 
 
 download_temp_path: "C:/Temp/"
 unzip_temp_path: "C:/Temp/kubernetes/"

ansible/roles/windows/kubernetes-node/tasks/main.yml

@@ -13,7 +13,7 @@
 
 - name: Ensure that any old instance of kubelet and kube-proxy is stopped
   win_service:
-    name: flanneld
+    name: "{{ item }}"
     state: stopped
     force_dependent_services: yes
   with_items:
@@ -212,7 +212,7 @@
         stderr_file: "{{ kube_proxy_log_path }}/{{ kube_proxy_service_name }}-stderr.log"
         state: present
         start_mode: auto
-        #dependencies: kubelet,docker  # latest NSSM throws a lot of errors when used with ansible and multiple dependencies
+        dependencies: "{{ kubelet_service_name }}"  # latest NSSM throws a lot of errors when used with ansible and multiple dependencies
 
     - name: Set kube-proxy service restart delay for short executions
       win_shell: "nssm set {{ kube_proxy_service_name }} AppThrottle 1500"
@@ -221,19 +221,19 @@
       win_shell: "nssm set {{ kube_proxy_service_name }} AppExit Default Restart"
 
     - name: Set kube-proxy service restart delay in case of failure
-      win_shell: "nssm set {{ kube_proxy_service_name }} AppRestartDelay 1000"
+      win_shell: "nssm set {{ kube_proxy_service_name }} AppRestartDelay 2000"
 
     - name: Set kube-proxy service to not overwrite stdout file
       win_shell: "nssm set {{ kube_proxy_service_name }} AppStdoutCreationDisposition 4"
 
     - name: Set kube-proxy service to not overwrite stderr file
       win_shell: "nssm set {{ kube_proxy_service_name }} AppStderrCreationDisposition 4"
 
-    - name: Set kube-proxy service dependency on docker
-      win_shell: "nssm set {{ kube_proxy_service_name }} DependOnService :Docker"
+    #- name: Set kube-proxy service dependency on docker
+    #  win_shell: "nssm set {{ kube_proxy_service_name }} DependOnService :Docker"
 
-    - name: Set kube-proxy service dependency on kubelet
-      win_shell: "nssm set {{ kube_proxy_service_name }} DependOnService +kubelet"
+    #- name: Set kube-proxy service dependency on kubelet
+    #  win_shell: "nssm set {{ kube_proxy_service_name }} DependOnService +kubelet"
 
 - name: Copy cleanup scripts
   win_copy:

ansible/vars_sample/main.yml

@@ -1,5 +1,7 @@
 ---
-# These are old sample without kubespray integration, see inventory/group_vars for more information
+# These are old sample without kubespray integration, see inventory/group_vars for updates structure
+# do not use it for kubespray integration.
+# Check inventory/group_vars for updated information
 kubernetes_master_ip: "{{ hostvars[groups['master-ubuntu'][0]]['ansible_default_ipv4']['address'] }}"
 kubernetes_pod_network_cidr: "10.200.0.0/16"
 kubernetes_cluster_dns: "10.201.0.10"