Liveness probe failed: private network connection refused

[JWDobken]

I am running my cluster with:

• ubuntu-20.04
• Kubernetes v1.18.8
• private network on IP range 10.244.0.0/16
• created the secrets for hcloud-csi
• flannel: https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
• controller-manager: https://raw.githubusercontent.com/hetznercloud/hcloud-cloud-controller-manager/master/deploy/ccm-networks.yaml

and that is all working fine:

Now when I deploy the CSI driver:

$ kubectl apply -f https://raw.githubusercontent.com/hetznercloud/csi-driver/master/deploy/kubernetes/hcloud-csi.yml
csidriver.storage.k8s.io/csi.hetzner.cloud created
storageclass.storage.k8s.io/hcloud-volumes created
serviceaccount/hcloud-csi created
clusterrole.rbac.authorization.k8s.io/hcloud-csi created
clusterrolebinding.rbac.authorization.k8s.io/hcloud-csi created
statefulset.apps/hcloud-csi-controller created
daemonset.apps/hcloud-csi-node created
service/hcloud-csi-controller-metrics created
service/hcloud-csi-node-metrics created

The hcloud-csi-driver of the hcloud-csi-controller and the hcloud-csi-nodes will keep restarting forever. But when I try to read the logs, I get:

$ kubectl -n kube-system logs hcloud-csi-controller-0 hcloud-csi-driver
Error from server: Get https://10.244.0.5:10250/containerLogs/kube-system/hcloud-csi-controller-0/hcloud-csi-driver: dial tcp 10.244.0.5:10250: connect: connection refused

and the events from the pod description reports that the Liveness probe fails:

Liveness probe failed: Get http://10.244.1.8:9808/healthz: dial tcp 10.244.1.8:9808: connect: connection refused

[JWDobken]

this solves my issue: #143 (comment)

[LKaemmerling]

@JWDobken we won't recommend using the host network for this. It looks like something on your network stack is wrong (Maybe missing routes?). Exposing the CSI Driver via the host network, exposes all the ports completely, so an attacker might use these open ports as an entry point.

[JWDobken]

thanks @LKaemmerling.

Also, I cannot shell connect to containers other than on the master node. So that might have the same cause. Although I can't figure out exactly what.

$ kubectl exec -it my-csi-app -- /bin/sh
Error from server: error dialing backend: dial tcp 10.244.0.3:10250: connect: no route to host

[JWDobken]

OK.. if I diable ufw on all nodes, everything works fine.

I'm definitely not an expert in this field at all, so if anyone can tell me how to properly set up my firewall for this private network it would be much appreciated.

My current ufw settings:

 ufw --force reset
ufw allow ssh
ufw allow 6443
ufw allow http
ufw allow https
ufw default deny incoming
ufw --force enable
ufw status verbose

[LKaemmerling]

OK.. if I diable ufw on all nodes, everything works fine.

I'm definitely not an expert in this field at all, so if anyone can tell me how to properly set up my firewall for this private network it would be much appreciated.

My current ufw settings:

 ufw --force reset
ufw allow ssh
ufw allow 6443
ufw allow http
ufw allow https
ufw default deny incoming
ufw --force enable
ufw status verbose

@JWDobken you should whitelist the whole private network range. As a sample

ufw allow from 10.0.0.0/8 # when this is your Hetzner Cloud Network Range

So basically you need to make sure that the intercluster communication can work.