vxlan keeps failing when trying to join network

[gimlichael]

So, after a whole weekend of trial and error with the "official" guide from Microsoft (https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/network-topologies) I decided (more or less by coincidence) to give this a try (refereed from https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-nodes/).

After my third re-install (which should have been the charm) i keep being blocked by a network issue.

It is running on a hypervisor having the en_windows_server_version_1909_updated_feb_2020_x64_dvd_8c906407.iso (Windows Server Version 1909) and an Intel NUC I7.

So, after the install, which followed this modified script:

{
    "Cri" : {
        "Name" : "dockerd",
        "Images" : {
            "Pause" : "mcr.microsoft.com/oss/kubernetes/pause:1.3.0",
            "Nanoserver" : "mcr.microsoft.com/windows/nanoserver:1909",
            "ServerCore" : "mcr.microsoft.com/windows/servercore:1909"
        }
    },
    "Cni" : {
        "Name" : "flannel",
        "Source" : [{ 
            "Name" : "flanneld",
            "Url" : "https://github.com/coreos/flannel/releases/download/v0.11.0/flanneld.exe"
            }
        ],
        "Plugin" : {
            "Name": "vxlan"
        },
        "InterfaceName" : "Ethernet"
    },
    "Kubernetes" : {
        "Source" : {
            "Release" : "1.17.3",
            "Url" : "https://dl.k8s.io/v1.17.3/kubernetes-node-windows-amd64.tar.gz"
        },
        "ControlPlane" : {
            "IpAddress" : "1.2.3.4",
            "Username" : "a-username",
            "KubeadmToken" : "a-token",
            "KubeadmCAHash" : "sha256:a-256-hash"
        },
        "KubeProxy" : {
            "Gates" : "WinOverlay=true"
        },
        "Network" : {
            "ServiceCidr" : "10.96.0.0/12",
            "ClusterCidr" : "10.244.0.0/16"
        }
    },
    "Install" : {
        "Destination" : "C:\\ProgramData\\Kubernetes"
    }
}

Everything supposedly goes fine until the flanneld is firing up:

C:\ProgramData\Kubernetes\flanneld.exe FlannelD()
OnStart - Successfully started the service 
I0309 23:48:55.743871    3408 main.go:450] Searching for interface using 1.2.3.46
I0309 23:48:55.898487    3408 main.go:210] Could not find valid interface matching 1.2.3.46: error looking up interface 1.2.3.46: Interface not found: 1.2.3.46
E0309 23:48:55.898487    3408 main.go:234] Failed to find interface to use that matches the interfaces and/or regexes provided


C:\ProgramData\Kubernetes\flanneld.exe exited unexpectedly 1

And the kubelet gives these errors:

Running on machine: k8s-n-w2-win
Binary: Built with gc go1.13.6 for windows/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0309 23:48:52.905969     700 server.go:749] kubelet needs to run as uid `0`. It is being run as -1
E0309 23:48:53.012433     700 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
E0309 23:48:53.257241     700 processstarttime.go:48] Could not get process start time, could not read /proc: CreateFile /proc: The system cannot find the file specified.
E0309 23:48:53.257241     700 processstarttime.go:48] Could not get process start time, could not read /proc: CreateFile /proc: The system cannot find the file specified.
E0309 23:48:53.258240     700 kubelet.go:1844] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
E0309 23:48:53.261240     700 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
E0309 23:48:53.269244     700 eviction_manager.go:246] eviction manager: failed to get summary stats: failed to get node info: node "k8s-n-w2-win" not found
E0309 23:48:53.288039     700 controller.go:227] failed to get node "k8s-n-w2-win" when trying to set owner ref to the node lease: nodes "k8s-n-w2-win" not found
E0309 23:48:53.359737     700 kubelet.go:2263] node "k8s-n-w2-win" not found
E0309 23:48:53.461269     700 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
E0309 23:48:55.908379     700 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.RuntimeClass: Get https://k8s:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:55.908379     700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://k8s:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dk8s-n-w2-win&limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:55.908379     700 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.CSIDriver: Get https://k8s:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:55.908379     700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://k8s:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:55.909379     700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://k8s:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-n-w2-win&limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:56.908606     700 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.RuntimeClass: Get https://k8s:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:56.910580     700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://k8s:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dk8s-n-w2-win&limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:56.911582     700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://k8s:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-n-w2-win&limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:56.912583     700 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.CSIDriver: Get https://k8s:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:56.913588     700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://k8s:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:57.916022     700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://k8s:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:57.916022     700 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.RuntimeClass: Get https://k8s:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:57.917016     700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://k8s:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dk8s-n-w2-win&limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:57.917016     700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://k8s:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-n-w2-win&limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:57.917016     700 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.CSIDriver: Get https://k8s:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:58.268916     700 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

I have disabled the firewall completely.

Linux is working perfectly.

Here is a kubectl get pods -n=kube-system

coredns-6955765f44-ndcnk          1/1     Running   0          3h56m
coredns-6955765f44-zt8qj          1/1     Running   0          3h56m
etcd-k8s-m-m                      1/1     Running   0          3h53m
etcd-k8s-n-m                      1/1     Running   0          3h52m
etcd-k8s-t-m                      1/1     Running   0          3h56m
kube-apiserver-k8s-m-m            1/1     Running   0          3h53m
kube-apiserver-k8s-n-m            1/1     Running   0          3h52m
kube-apiserver-k8s-t-m            1/1     Running   0          3h56m
kube-controller-manager-k8s-m-m   1/1     Running   1          3h53m
kube-controller-manager-k8s-n-m   1/1     Running   0          3h52m
kube-controller-manager-k8s-t-m   1/1     Running   1          3h56m
kube-flannel-ds-amd64-2ptrt       1/1     Running   0          152m
kube-flannel-ds-amd64-pb7dl       1/1     Running   1          118m
kube-flannel-ds-amd64-rmfp2       1/1     Running   0          152m
kube-flannel-ds-amd64-szpbk       1/1     Running   0          153m
kube-flannel-ds-amd64-vsdc9       1/1     Running   2          118m
kube-flannel-ds-amd64-zfnd5       1/1     Running   1          118m
kube-proxy-78w2f                  1/1     Running   1          3h53m
kube-proxy-7bfzv                  1/1     Running   1          118m
kube-proxy-cfl6z                  1/1     Running   0          3h56m
kube-proxy-jw8kw                  1/1     Running   0          3h52m
kube-proxy-m8flw                  1/1     Running   1          118m
kube-proxy-qvb52                  1/1     Running   1          118m
kube-scheduler-k8s-m-m            1/1     Running   0          3h53m
kube-scheduler-k8s-n-m            1/1     Running   0          3h52m
kube-scheduler-k8s-t-m            1/1     Running   1          3h56m

Here is the kube-flannel.yaml

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
    - configMap
    - secret
    - emptyDir
    - hostPath
  allowedHostPaths:
    - pathPrefix: "/etc/cni/net.d"
    - pathPrefix: "/etc/kube-flannel"
    - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups: ['extensions']
    resources: ['podsecuritypolicies']
    verbs: ['use']
    resourceNames: ['psp.flannel.unprivileged']
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "vxlan0",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan",
        "VNI" : 4096,
        "Port": 4789
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - amd64
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.11.0-amd64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.11.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

Any help is greatly appreciated.

When I do a kubectl get nodes, it is listing the win worker .. but given the errors listed above, i have no hope that it is working.

NAME           STATUS   ROLES    AGE     VERSION
k8s-m-m        Ready    master   3h31m   v1.17.3
k8s-m-w1       Ready    <none>   96m     v1.17.3
k8s-n-m        Ready    master   3h30m   v1.17.3
k8s-n-w1       Ready    <none>   96m     v1.17.3
k8s-n-w2-win   Ready    <none>   6m43s   v1.17.3
k8s-t-m        Ready    master   3h34m   v1.17.3
k8s-t-w1       Ready    <none>   96m     v1.17.3

So until i hear from a smart person, I have applied the -reset to the scripts.

Thanks.

[gimlichael]

I gave it an extra shot today, and a search somehow got me to this issue on flannel: flannel-io/flannel#1178

It seems to me, that the reason for my errors is a result of everything going so fast, that documentation is not kept up to date.

For instance, in the lastest kube-flannel.yaml, name is cbr0.
However, in the script, it is vxlan0.

Changing this be simply re-applying kube-flannel does in fact change the masters /etc/cni/net.d/10-flannel.conflist.

You really need to keep an eye out for these small tweaks.

For now it seems i can join - and unlike my two previous attempts (host-gw and vxlan) it also looks like I have connectivity from the POD.

A small hooray for now. Hope this can help someone struggling the win-worker challenge ;-)

[gimlichael]

There is most defiantly a bug somewhere .. sometime it takes 5+ retries to do the -install, -join and -reset (repeat), before the node is actually joined.

Getting tired - just wanted to keep you posted ..

[gimlichael]

I really don't know what is up and down .. sometime it works .. sometime not .. sometime some of the nodes .. sometimes others ..

Here is an error from one of the nodes:

C:\ProgramData\Kubernetes\flanneld.exe FlannelD()
OnStart - Successfully started the service 
I0311 02:30:56.098822    1988 main.go:450] Searching for interface using 192.168.1.212
I0311 02:30:56.197861    1988 main.go:527] Using interface with name vEthernet (Ethernet) and address 192.168.1.212
I0311 02:30:56.197861    1988 main.go:544] Defaulting external address to interface address (192.168.1.212)
I0311 02:30:56.199861    1988 kube.go:126] Waiting 10m0s for node controller to sync
I0311 02:30:56.199861    1988 kube.go:309] Starting kube subnet manager
I0311 02:30:57.208472    1988 kube.go:133] Node controller sync successful
I0311 02:30:57.208652    1988 main.go:244] Created subnet manager: Kubernetes Subnet Manager - k8s-t-w2-win
I0311 02:30:57.208652    1988 main.go:247] Installing signal handlers
I0311 02:30:57.208652    1988 main.go:386] Found network config - Backend type: vxlan
I0311 02:30:57.208652    1988 vxlan_windows.go:123] VXLAN config: Name=vxlan0 MacPrefix=0E-2A VNI=4096 Port=4789 GBP=false DirectRouting=false
I0311 02:30:57.215650    1988 vxlan_windows.go:133] Checking HNS network for DR MAC : [{Id:3c6fab4d-6191-4171-a627-dd46f8c2e392 Name:External Type:Overlay Policies:[{Type:NetAdapterName Settings:[123 34 78 101 116 119 111 114 107 65 100 97 112 116 101 114 78 97 109 101 34 58 34 69 116 104 101 114 110 101 116 34 125]} {Type:VSwitchExtension Settings:[123 34 69 120 116 101 110 115 105 111 110 73 68 34 58 34 101 55 99 51 98 50 102 48 45 102 51 99 53 45 52 56 100 102 45 97 102 50 98 45 49 48 102 101 100 54 100 55 50 101 55 97 34 125]} {Type:VSwitchExtension Settings:[123 34 69 120 116 101 110 115 105 111 110 73 68 34 58 34 101 57 98 53 57 99 102 97 45 50 98 101 49 45 52 98 50 49 45 56 50 56 102 45 98 54 102 98 100 98 100 100 99 48 49 55 34 44 34 69 110 97 98 108 101 34 58 116 114 117 101 125]} {Type:VSwitchExtension Settings:[123 34 69 120 116 101 110 115 105 111 110 73 68 34 58 34 101 97 50 52 99 100 54 99 45 100 49 55 97 45 52 51 52 56 45 57 49 57 48 45 48 57 102 48 100 53 98 101 56 51 100 100 34 125]} {Type:ProviderAddress Settings:[123 34 80 114 111 118 105 100 101 114 65 100 100 114 101 115 115 34 58 34 49 57 50 46 49 54 56 46 49 46 50 49 50 34 125]} {Type:DrMacAddress Settings:[123 34 65 100 100 114 101 115 115 34 58 34 48 48 45 49 53 45 53 68 45 70 57 45 52 49 45 67 54 34 125]}] MacPool:{Ranges:[{StartMacAddress:00-15-5D-37-A0-00 EndMacAddress:00-15-5D-37-AF-FF}]} Dns:{Suffix: Search:[] ServerList:[] Options:[]} Ipams:[{Type: Subnets:[{IpAddressPrefix:192.168.255.0/30 Policies:[[123 34 84 121 112 101 34 58 34 86 83 73 68 34 44 34 83 101 116 116 105 110 103 115 34 58 123 34 73 115 111 108 97 116 105 111 110 73 100 34 58 57 57 57 57 125 125]] Routes:[{NextHop:192.168.255.1 DestinationPrefix:0.0.0.0/0 Metric:0}]}]}] Flags:0 SchemaVersion:{Major:2 Minor:0}}]
I0311 02:30:57.216839    1988 vxlan_windows.go:133] Checking HNS network for DR MAC : [{Id:6b9df2ca-9769-4f7e-8f58-696d852dc261 Name:nat Type: Policies:[{Type:VSwitchExtension Settings:[123 34 69 120 116 101 110 115 105 111 110 73 68 34 58 34 101 55 99 51 98 50 102 48 45 102 51 99 53 45 52 56 100 102 45 97 102 50 98 45 49 48 102 101 100 54 100 55 50 101 55 97 34 125]} {Type:VSwitchExtension Settings:[123 34 69 120 116 101 110 115 105 111 110 73 68 34 58 34 101 57 98 53 57 99 102 97 45 50 98 101 49 45 52 98 50 49 45 56 50 56 102 45 98 54 102 98 100 98 100 100 99 48 49 55 34 125]} {Type:VSwitchExtension Settings:[123 34 69 120 116 101 110 115 105 111 110 73 68 34 58 34 101 97 50 52 99 100 54 99 45 100 49 55 97 45 52 51 52 56 45 57 49 57 48 45 48 57 102 48 100 53 98 101 56 51 100 100 34 125]}] MacPool:{Ranges:[{StartMacAddress:00-15-5D-39-50-00 EndMacAddress:00-15-5D-39-5F-FF}]} Dns:{Suffix: Search:[] ServerList:[] Options:[]} Ipams:[{Type: Subnets:[{IpAddressPrefix:172.17.208.0/20 Policies:[[123 34 84 121 112 101 34 58 34 86 76 65 78 34 44 34 83 101 116 116 105 110 103 115 34 58 123 34 73 115 111 108 97 116 105 111 110 73 100 34 58 49 125 125]] Routes:[{NextHop:172.17.208.1 DestinationPrefix:0.0.0.0/0 Metric:0}]}]}] Flags:8 SchemaVersion:{Major:2 Minor:0}}]
I0311 02:30:57.234817    1988 device_windows.go:103] Attempting to create HNSNetwork {"Name":"vxlan0","Type":"Overlay","Subnets":[{"AddressPrefix":"10.244.12.0/24","GatewayAddress":"10.244.12.1","Policies":[{"Type":"VSID","VSID":4096}]}]}
E0311 02:30:57.279352    1988 main.go:289] Error registering network: failed to create HNSNetwork vxlan0: hnsCall failed in Win32: The parameter is incorrect. (0x57)
I0311 02:30:57.283618    1988 main.go:366] Stopping shutdownHandler...


C:\ProgramData\Kubernetes\flanneld.exe exited unexpectedly 1

[gimlichael]

After a restart of the 3 worker nodes:

C:\ProgramData\Kubernetes\flanneld.exe FlannelD()
OnStart - Successfully started the service 
I0311 02:52:17.464429    3736 main.go:450] Searching for interface using 192.168.1.212
I0311 02:52:17.577710    3736 main.go:210] Could not find valid interface matching 192.168.1.212: error looking up interface 192.168.1.212: Interface not found: 192.168.1.212
E0311 02:52:17.577710    3736 main.go:234] Failed to find interface to use that matches the interfaces and/or regexes provided


C:\ProgramData\Kubernetes\flanneld.exe exited unexpectedly 1

C:\ProgramData\Kubernetes\flanneld.exe FlannelD()
OnStart - Successfully started the service 
I0311 02:52:15.218383    3796 main.go:450] Searching for interface using 192.168.1.216
I0311 02:52:15.337120    3796 main.go:210] Could not find valid interface matching 192.168.1.216: error looking up interface 192.168.1.216: Interface not found: 192.168.1.216
E0311 02:52:15.337120    3796 main.go:234] Failed to find interface to use that matches the interfaces and/or regexes provided


C:\ProgramData\Kubernetes\flanneld.exe exited unexpectedly 1

C:\ProgramData\Kubernetes\flanneld.exe FlannelD()
OnStart - Successfully started the service 
I0311 02:51:54.311767    3828 main.go:450] Searching for interface using 192.168.1.214
I0311 02:51:54.469993    3828 main.go:527] Using interface with name vEthernet (Ethernet) and address 192.168.1.214
I0311 02:51:54.469993    3828 main.go:544] Defaulting external address to interface address (192.168.1.214)
I0311 02:51:54.481993    3828 kube.go:309] Starting kube subnet manager
I0311 02:51:54.481993    3828 kube.go:126] Waiting 10m0s for node controller to sync
I0311 02:51:55.495856    3828 kube.go:133] Node controller sync successful
I0311 02:51:55.495856    3828 main.go:244] Created subnet manager: Kubernetes Subnet Manager - k8s-m-w2-win
I0311 02:51:55.495856    3828 main.go:247] Installing signal handlers
I0311 02:51:55.495856    3828 main.go:386] Found network config - Backend type: vxlan
I0311 02:51:55.495856    3828 vxlan_windows.go:123] VXLAN config: Name=vxlan0 MacPrefix=0E-2A VNI=4096 Port=4789 GBP=false DirectRouting=false
I0311 02:51:55.499855    3828 vxlan_windows.go:133] Checking HNS network for DR MAC : [{Id:73052dcb-f8ff-4015-865e-b05aecbd83bd Name:External Type:Overlay Policies:[{Type:NetAdapterName Settings:[123 34 78 101 116 119 111 114 107 65 100 97 112 116 101 114 78 97 109 101 34 58 34 69 116 104 101 114 110 101 116 34 125]} {Type:VSwitchExtension Settings:[123 34 69 120 116 101 110 115 105 111 110 73 68 34 58 34 101 55 99 51 98 50 102 48 45 102 51 99 53 45 52 56 100 102 45 97 102 50 98 45 49 48 102 101 100 54 100 55 50 101 55 97 34 125]} {Type:VSwitchExtension Settings:[123 34 69 120 116 101 110 115 105 111 110 73 68 34 58 34 101 57 98 53 57 99 102 97 45 50 98 101 49 45 52 98 50 49 45 56 50 56 102 45 98 54 102 98 100 98 100 100 99 48 49 55 34 44 34 69 110 97 98 108 101 34 58 116 114 117 101 125]} {Type:VSwitchExtension Settings:[123 34 69 120 116 101 110 115 105 111 110 73 68 34 58 34 101 97 50 52 99 100 54 99 45 100 49 55 97 45 52 51 52 56 45 57 49 57 48 45 48 57 102 48 100 53 98 101 56 51 100 100 34 125]} {Type:ProviderAddress Settings:[123 34 80 114 111 118 105 100 101 114 65 100 100 114 101 115 115 34 58 34 49 57 50 46 49 54 56 46 49 46 50 49 52 34 125]} {Type:DrMacAddress Settings:[123 34 65 100 100 114 101 115 115 34 58 34 48 48 45 49 53 45 53 68 45 54 67 45 54 66 45 70 65 34 125]}] MacPool:{Ranges:[{StartMacAddress:00-15-5D-1C-90-00 EndMacAddress:00-15-5D-1C-9F-FF}]} Dns:{Suffix: Search:[] ServerList:[] Options:[]} Ipams:[{Type: Subnets:[{IpAddressPrefix:192.168.255.0/30 Policies:[[123 34 84 121 112 101 34 58 34 86 83 73 68 34 44 34 83 101 116 116 105 110 103 115 34 58 123 34 73 115 111 108 97 116 105 111 110 73 100 34 58 57 57 57 57 125 125]] Routes:[{NextHop:192.168.255.1 DestinationPrefix:0.0.0.0/0 Metric:0}]}]}] Flags:0 SchemaVersion:{Major:2 Minor:0}}]
I0311 02:51:55.499855    3828 vxlan_windows.go:133] Checking HNS network for DR MAC : [{Id:cf10693e-20b1-401e-9d08-a56d37ec154c Name:nat Type: Policies:[{Type:VSwitchExtension Settings:[123 34 69 120 116 101 110 115 105 111 110 73 68 34 58 34 101 55 99 51 98 50 102 48 45 102 51 99 53 45 52 56 100 102 45 97 102 50 98 45 49 48 102 101 100 54 100 55 50 101 55 97 34 125]} {Type:VSwitchExtension Settings:[123 34 69 120 116 101 110 115 105 111 110 73 68 34 58 34 101 57 98 53 57 99 102 97 45 50 98 101 49 45 52 98 50 49 45 56 50 56 102 45 98 54 102 98 100 98 100 100 99 48 49 55 34 125]} {Type:VSwitchExtension Settings:[123 34 69 120 116 101 110 115 105 111 110 73 68 34 58 34 101 97 50 52 99 100 54 99 45 100 49 55 97 45 52 51 52 56 45 57 49 57 48 45 48 57 102 48 100 53 98 101 56 51 100 100 34 125]}] MacPool:{Ranges:[{StartMacAddress:00-15-5D-A2-80-00 EndMacAddress:00-15-5D-A2-8F-FF}]} Dns:{Suffix: Search:[] ServerList:[] Options:[]} Ipams:[{Type: Subnets:[{IpAddressPrefix:172.30.48.0/20 Policies:[[123 34 84 121 112 101 34 58 34 86 76 65 78 34 44 34 83 101 116 116 105 110 103 115 34 58 123 34 73 115 111 108 97 116 105 111 110 73 100 34 58 49 125 125]] Routes:[{NextHop:172.30.48.1 DestinationPrefix:0.0.0.0/0 Metric:0}]}]}] Flags:8 SchemaVersion:{Major:2 Minor:0}}]
I0311 02:51:55.510856    3828 device_windows.go:103] Attempting to create HNSNetwork {"Name":"vxlan0","Type":"Overlay","Subnets":[{"AddressPrefix":"10.244.5.0/24","GatewayAddress":"10.244.5.1","Policies":[{"Type":"VSID","VSID":4096}]}]}
E0311 02:51:55.539569    3828 main.go:289] Error registering network: failed to create HNSNetwork vxlan0: hnsCall failed in Win32: The parameter is incorrect. (0x57)
I0311 02:51:55.550989    3828 main.go:366] Stopping shutdownHandler...


C:\ProgramData\Kubernetes\flanneld.exe exited unexpectedly 1

[benmoss]

As part of the move to the new kubeadm on Windows approach with 1.18 we are closing issues related to the previous alpha.
/close

[k8s-ci-robot]

@benmoss: Closing this issue.

In response to this:

As part of the move to the new kubeadm on Windows approach with 1.18 we are closing issues related to the previous alpha.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gimlichael]

As part of the move to the new kubeadm on Windows approach with 1.18 we are closing issues related to the previous alpha.
/close

Sounds good; I am actually pursuing this now - however, my deamonsets are not working and the latest 0.12.0r2 Flannel image is raising errors. Do you have some reading material to follow?

Currently I am running 1.17.3.

Thanks.

[benmoss]

Docs are available here: https://deploy-preview-19217--kubernetes-io-vnext-staging.netlify.com/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/

[gimlichael]

Thank you for the link; i can see I worked most things out yesterday by browsing this tool. That said, even following the new documentation, I am experiencing errors from both DeamonSet.

I will, until guided otherwise, provide the error information here (hopefully it can help you guys out):

FLANNEL (sigwindowstools/flannel:0.12.0-rc2)

Error: failed to start container "kube-flannel": Error response from daemon: hcsshim::CreateComputeSystem kube-flannel: The system cannot find the file specified. (extra info: {"SystemType":"Container","Name":"kube-flannel","Owner":"docker","VolumePath":"\\\\?\\Volume{2054e65a-54cf-4cd8-aa05-c0b26a52c8c1}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\kube-flannel","Layers":[{"ID":"bba4b654-1653-51fd-aafa-ca4002ab3764","Path":"C:\\ProgramData\\docker\\windowsfilter\\fd303b82e5567b37c8fd02ddc364130b3999bd7c50959f69664777ae5948a80f"},{"ID":"d62f3c52-15d7-5d3a-be3a-6204d6ce2180","Path":"C:\\ProgramData\\docker\\windowsfilter\\808187018f42058c88d1f39510e83bf6fc88d4ca258e377a3d2a561503a3cb14"},{"ID":"94d4df10-24d9-5933-96e7-97bc618f64c4","Path":"C:\\ProgramData\\docker\\windowsfilter\\955a011dc88acdb459566fca09214ff6830af3405dfce4df98e622a1bfedc002"},{"ID":"a78c70f4-6d7f-57be-97ca-cc4746cc93c9","Path":"C:\\ProgramData\\docker\\windowsfilter\\f2ac0b1c6334f6b057b75adb5820d6298405140b913dbbb3fe50c99e0e796080"},{"ID":"26bea7b7-e0c9-5f0b-9ed5-02649a726cbe","Path":"C:\\ProgramData\\docker\\windowsfilter\\1c549b7c67400c55fff6263c32fb87a78ebeb2178366ac4a0bf37c542918699c"},{"ID":"8ddacbfd-6272-52e3-aa91-9c2b2df40045","Path":"C:\\ProgramData\\docker\\windowsfilter\\7b7930f47395765a9f9834a41e1c2753c20e4ac9d61eb8410dc1a67b4df7be12"},{"ID":"89b9b5d2-3a36-5fb2-be0b-86a12bb65d00","Path":"C:\\ProgramData\\docker\\windowsfilter\\1a4a1dc83aeabea013646dcb2603d0df538825004da1818b24848a104c5bbb13"},{"ID":"4f8a75c7-e09c-5052-9199-fd1334de8275","Path":"C:\\ProgramData\\docker\\windowsfilter\\37a4063e4a7cf00cfae0f7b333ebc930ce46e6cfc97ac20e73d07d9b4f58951e"},{"ID":"cee98c63-0a52-5bfc-9e43-46fe202a8d91","Path":"C:\\ProgramData\\docker\\windowsfilter\\52c7e342d4f3e4ab80ee5862dcd95561fd23502a3a04cbfaf65bcd743da4aacd"}],"ProcessorWeight":5000,"HostName":"aedfeebf61d8","MappedDirectories":[{"HostPath":"c:\\","ContainerPath":"c:\\host","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\var\\lib\\kubelet\\pods\\cdf872b1-3258-4098-951a-1b195a22a228\\volumes\\kubernetes.io~configmap\\kube-proxy","ContainerPath":"c:\\kube-proxy","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\var\\lib\\kubelet\\pods\\cdf872b1-3258-4098-951a-1b195a22a228\\volumes\\kubernetes.io~configmap\\flannel-windows-cfg","ContainerPath":"c:\\etc\\kube-flannel-windows","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\var\\lib\\kubelet\\pods\\cdf872b1-3258-4098-951a-1b195a22a228\\volumes\\kubernetes.io~configmap\\kubeadm-config","ContainerPath":"c:\\etc\\kubeadm-config","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\var\\lib\\kubelet\\pods\\cdf872b1-3258-4098-951a-1b195a22a228\\volumes\\kubernetes.io~configmap\\flannel-cfg","ContainerPath":"c:\\etc\\kube-flannel","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\etc","ContainerPath":"c:\\etc\\cni\\net.d","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\var\\lib\\kubelet\\pods\\cdf872b1-3258-4098-951a-1b195a22a228\\volumes\\kubernetes.io~secret\\flannel-token-dm4w5","ContainerPath":"c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false}],"MappedPipes":[{"HostPath":"\\\\.\\pipe\\rancher_wins","ContainerPipeName":"rancher_wins"}],"HvPartition":false,"NetworkSharedContainerName":"aedfeebf61d8357bc3bf2b0d4779bc38e951b818278acd56e6f55f342f34ca3b","EndpointList":["91EDD84B-299F-4B76-85AF-7F8598B0CA03"]})

KUBEPROXY (sigwindowstools/kube-proxy:v1.17.3)

network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

The error from kube-proxy make sense, given that Flannel is not setup .. i am guessing that Flannel is also providing the HNS network?

Thanks.

[benmoss]

Are you running wins? You can't just run the DaemonSets without wins as well.

[gimlichael]

Basically I followed the document and started with the preparenode PowerShell. It looks like it is doing some wins setup using rancher. Anything I overlooked?

[benmoss]

Hmm, I don't know what file it would be talking about then. I thought it was the HostPath volume mount.