Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CrashLoopBackOff for hcloud-csi-controller with "Failed to connect to the CSI driver" #742

Open
wanGiB opened this issue Oct 7, 2024 · 7 comments
Labels
question Further information is requested

Comments

@wanGiB
Copy link

wanGiB commented Oct 7, 2024

Environment:

Kubernetes version: 1.2.9
Hetzner Cloud CSI Driver version: latest
Pod affected: hcloud-csi-controller
Namespace: kube-system

Problem:

I am facing an issue where the hcloud-csi-controller pod goes into CrashLoopBackOff. 4 of the containers are consistently crashing, and logs show the following error message:

W1007 18:45:39.817729       1 connection.go:173] Still connecting to unix:///run/csi/socket
W1007 18:45:49.818479       1 connection.go:173] Still connecting to unix:///run/csi/socket
E1007 18:21:31.028816       1 main.go:149] "Failed to connect to the CSI driver" err="context deadline exceeded" csiAddress="/run/csi/socket"

The pod status indicates that only 1 out of 5 containers are running, with 4 stuck in a crash loop.

kube-system    hcloud-cloud-controller-manager-865cf8bf5d-p2wgm       1/1     Running            0                61m
kube-system    hcloud-csi-controller-55d4676fb9-z6d4w                 1/5     CrashLoopBackOff   25 (2m32s ago)   11m
kube-system    hcloud-csi-node-5hzm8                                  3/3     Running            0                11m
kube-system    hcloud-csi-node-dq2xb                                  3/3     Running            0                11m
kube-system    hcloud-csi-node-gk9q4                                  3/3     Running            0                11m

Setup:
This is how I set up the CSI and Cloud Controller Manager:

kubectl apply -f https://raw.githubusercontent.com/hetznercloud/csi-driver/main/deploy/kubernetes/hcloud-csi.yml
kubectl apply -f https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/latest/download/ccm.yaml

@lukasmetzner
Copy link
Contributor

Hi,

to assist you with your issue, I’ll need a bit more information.

  • Are you running Kubernetes version 1.2.9 or did you meant to write 1.29?
  • Did you configure the secret for the hcloud token as described in: Kubernetes - Getting Started
  • Could you provide the output of kubectl -n kube-system describe pod hcloud-csi-controller-XXXXXXX-XXXXX?
  • Can you provide the logs of all containers in the pod: kubectl -n kube-system logs hcloud-csi-controller-XXXXXXX-XXXXX --all-containers --prefix --previous --timestamps

If you do not want to publicly share your logs, you can also open up a customer support ticket and reference this issue there. https://console.hetzner.cloud/support

Best Regards
Lukas

@lukasmetzner lukasmetzner added the question Further information is requested label Oct 8, 2024
@wanGiB
Copy link
Author

wanGiB commented Oct 8, 2024

Hi @lukasmetzner thanks for the reply ,

I am running Kubernetes version 1.29. This was the script I used with some unrelated parts omitted:

# Apply Hetzner Cloud Secrets
$KUBECTL apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: hcloud
  namespace: kube-system
stringData:
  token: "$hcloud_token"
EOF

# Apply Hetzner Cloud CSI and CCM manifests
kubectl apply -f https://raw.githubusercontent.com/hetznercloud/csi-driver/main/deploy/kubernetes/hcloud-csi.yml
kubectl apply -f https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/latest/download/ccm.yaml
kubectl -n kube-system patch ds kube-flannel-ds --type json -p '[{"op":"add","path":"/spec/template/spec/tolerations/-","value":{"key":"node.cloudprovider.kubernetes.io/uninitialized","value":"true","effect":"NoSchedule"}}]'

Here is the log output from all containers:

kubectl logs hcloud-csi-controller-7db6667bfc-7g46h -n kube-system --all-containers --prefix --timestamps:

[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-attacher] 2024-10-08T12:01:55.986768840Z I1008 12:01:55.986589       1 main.go:109] "Version" version="v4.7.0"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-attacher] 2024-10-08T12:02:05.989089555Z I1008 12:02:05.988941       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-attacher] 2024-10-08T12:02:15.989185908Z I1008 12:02:15.989032       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-attacher] 2024-10-08T12:02:25.988292103Z I1008 12:02:25.988080       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-attacher] 2024-10-08T12:02:25.988335104Z E1008 12:02:25.988217       1 main.go:149] "Failed to connect to the CSI driver" err="context deadline exceeded" csiAddress="/run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-resizer] 2024-10-08T12:01:56.046495272Z I1008 12:01:56.046292       1 main.go:108] "Version" version="v1.12.0"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-resizer] 2024-10-08T12:02:06.047961535Z I1008 12:02:06.047741       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-resizer] 2024-10-08T12:02:16.047703362Z I1008 12:02:16.047548       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-resizer] 2024-10-08T12:02:26.047869295Z I1008 12:02:26.047715       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-resizer] 2024-10-08T12:02:26.047978691Z E1008 12:02:26.047864       1 main.go:153] "Failed to create CSI client" err="failed to connect to CSI driver: context deadline exceeded"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-provisioner] 2024-10-08T12:01:57.095937459Z W1008 12:01:57.095680       1 feature_gate.go:354] Setting GA feature gate Topology=true. It will be removed in a future release.
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-provisioner] 2024-10-08T12:01:57.095998333Z I1008 12:01:57.095812       1 csi-provisioner.go:154] Version: v5.1.0
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-provisioner] 2024-10-08T12:01:57.096003012Z I1008 12:01:57.095819       1 csi-provisioner.go:177] Building kube configs for running in cluster...
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-provisioner] 2024-10-08T12:02:07.097487650Z I1008 12:02:07.097323       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-provisioner] 2024-10-08T12:02:17.096734539Z I1008 12:02:17.096560       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-provisioner] 2024-10-08T12:02:27.097401471Z I1008 12:02:27.097245       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/csi-provisioner] 2024-10-08T12:02:27.097549910Z E1008 12:02:27.097388       1 csi-provisioner.go:215] context deadline exceeded
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:31:02.716921280Z I1008 11:31:02.716731       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:31:12.716404262Z I1008 11:31:12.716204       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:31:22.717244851Z I1008 11:31:22.717069       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:31:32.716364211Z I1008 11:31:32.716192       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:31:42.716874770Z I1008 11:31:42.716727       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:31:52.717002762Z I1008 11:31:52.716829       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:32:02.716598105Z I1008 11:32:02.716428       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:32:12.716876108Z I1008 11:32:12.716688       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:32:22.717062350Z I1008 11:32:22.716868       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:32:32.716432570Z I1008 11:32:32.716277       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:32:42.716632517Z I1008 11:32:42.716436       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:32:52.716632919Z I1008 11:32:52.716441       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:33:02.717109285Z I1008 11:33:02.716930       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:33:12.716623155Z I1008 11:33:12.716432       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:33:22.716330578Z I1008 11:33:22.716140       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:33:32.716713689Z I1008 11:33:32.716535       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:33:42.716680218Z I1008 11:33:42.716527       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:33:52.716869314Z I1008 11:33:52.716643       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:34:02.716296341Z I1008 11:34:02.716138       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:34:12.717056759Z I1008 11:34:12.716903       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:34:22.716341300Z I1008 11:34:22.716141       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:34:32.716257674Z I1008 11:34:32.716105       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:34:42.717342311Z I1008 11:34:42.717194       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:34:52.716252939Z I1008 11:34:52.716105       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:35:02.716942494Z I1008 11:35:02.716798       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:35:12.716553156Z I1008 11:35:12.716426       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:35:22.717223576Z I1008 11:35:22.717049       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:35:32.716558770Z I1008 11:35:32.716368       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:35:42.716387000Z I1008 11:35:42.716234       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:35:52.717100510Z I1008 11:35:52.716917       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:36:02.716291555Z I1008 11:36:02.716105       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:36:12.716558607Z I1008 11:36:12.716367       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:36:22.717001089Z I1008 11:36:22.716845       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:36:32.717059791Z I1008 11:36:32.716909       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:36:42.717037772Z I1008 11:36:42.716864       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:36:52.716559937Z I1008 11:36:52.716404       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:37:02.717084924Z I1008 11:37:02.716907       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:37:12.716922291Z I1008 11:37:12.716733       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:37:22.716708932Z I1008 11:37:22.716522       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:37:32.716970395Z I1008 11:37:32.716800       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:37:42.717246996Z I1008 11:37:42.716925       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:37:52.716325890Z I1008 11:37:52.716166       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:38:02.716731973Z I1008 11:38:02.716576       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:38:12.717106407Z I1008 11:38:12.716934       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:38:22.716593166Z I1008 11:38:22.716392       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:38:32.716493080Z I1008 11:38:32.716308       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:38:42.717070305Z I1008 11:38:42.716945       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:38:52.716305833Z I1008 11:38:52.716120       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:39:02.716724871Z I1008 11:39:02.716512       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:39:12.716282152Z I1008 11:39:12.716129       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:39:22.716779217Z I1008 11:39:22.716579       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:39:32.717220717Z I1008 11:39:32.717026       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:39:42.716352359Z I1008 11:39:42.716162       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:39:52.717265254Z I1008 11:39:52.717111       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:40:02.716892065Z I1008 11:40:02.716732       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:40:12.716285490Z I1008 11:40:12.716117       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:40:22.716984894Z I1008 11:40:22.716786       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:40:32.717014170Z I1008 11:40:32.716842       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:40:42.716390552Z I1008 11:40:42.716182       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:40:52.716866166Z I1008 11:40:52.716692       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:41:02.717350376Z I1008 11:41:02.717164       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:41:12.716810135Z I1008 11:41:12.716571       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:41:22.716596065Z I1008 11:41:22.716421       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:41:32.716961552Z I1008 11:41:32.716802       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:41:42.716977535Z I1008 11:41:42.716811       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:41:52.716419089Z I1008 11:41:52.716243       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:42:02.716726357Z I1008 11:42:02.716546       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:42:12.716527025Z I1008 11:42:12.716344       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:42:22.716690624Z I1008 11:42:22.716516       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:42:32.716513112Z I1008 11:42:32.716339       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:42:42.716701789Z I1008 11:42:42.716520       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:42:52.716451310Z I1008 11:42:52.716289       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:43:02.716793805Z I1008 11:43:02.716626       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:43:12.716900015Z I1008 11:43:12.716735       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:43:22.716269044Z I1008 11:43:22.716138       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:43:32.716658867Z I1008 11:43:32.716490       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:43:42.717038100Z I1008 11:43:42.716864       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:43:52.716556238Z I1008 11:43:52.716386       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:44:02.716368718Z I1008 11:44:02.716179       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:44:12.716765595Z I1008 11:44:12.716555       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:44:22.716784702Z I1008 11:44:22.716601       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:44:32.716810603Z I1008 11:44:32.716656       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:44:42.717189175Z I1008 11:44:42.717002       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:44:52.716509301Z I1008 11:44:52.716316       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:45:02.716825767Z I1008 11:45:02.716593       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:45:12.716558597Z I1008 11:45:12.716396       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:45:22.717214159Z I1008 11:45:22.717028       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:45:32.716956127Z I1008 11:45:32.716796       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:45:42.716536281Z I1008 11:45:42.716341       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:45:52.716975808Z I1008 11:45:52.716832       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:46:02.716353071Z I1008 11:46:02.716179       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:46:12.717127997Z I1008 11:46:12.716942       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:46:22.717014225Z I1008 11:46:22.716834       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:46:32.717020499Z I1008 11:46:32.716846       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:46:42.716261717Z I1008 11:46:42.716119       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:46:52.716635701Z I1008 11:46:52.716465       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:47:02.717106175Z I1008 11:47:02.716943       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:47:12.716803490Z I1008 11:47:12.716621       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:47:22.716962259Z I1008 11:47:22.716804       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:47:32.717226757Z I1008 11:47:32.717043       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:47:42.716432158Z I1008 11:47:42.716293       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:47:52.716937768Z I1008 11:47:52.716771       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:48:02.716968508Z I1008 11:48:02.716801       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:48:12.717114584Z I1008 11:48:12.716952       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:48:22.716512978Z I1008 11:48:22.716325       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:48:32.717286450Z I1008 11:48:32.717078       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:48:42.716975689Z I1008 11:48:42.716801       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:48:52.716378030Z I1008 11:48:52.716213       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:49:02.716466658Z I1008 11:49:02.716290       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:49:12.716467351Z I1008 11:49:12.716294       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:49:22.717331384Z I1008 11:49:22.717168       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:49:32.716600043Z I1008 11:49:32.716421       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:49:42.716426199Z I1008 11:49:42.716245       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:49:52.716396275Z I1008 11:49:52.716243       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:50:02.716245584Z I1008 11:50:02.716111       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:50:12.716971467Z I1008 11:50:12.716808       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:50:22.716547975Z I1008 11:50:22.716349       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:50:32.717005004Z I1008 11:50:32.716840       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:50:42.717320307Z I1008 11:50:42.717159       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:50:52.716465596Z I1008 11:50:52.716256       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:51:02.716431815Z I1008 11:51:02.716264       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:51:12.716948626Z I1008 11:51:12.716796       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:51:22.716924393Z I1008 11:51:22.716799       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:51:32.716356058Z I1008 11:51:32.716180       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:51:42.716944404Z I1008 11:51:42.716787       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:51:52.716918888Z I1008 11:51:52.716777       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:52:02.716952352Z I1008 11:52:02.716800       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:52:12.716598871Z I1008 11:52:12.716380       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:52:22.716296275Z I1008 11:52:22.716111       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:52:32.716383851Z I1008 11:52:32.716224       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:52:42.716598947Z I1008 11:52:42.716415       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:52:52.717002967Z I1008 11:52:52.716826       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:53:02.717104448Z I1008 11:53:02.716938       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:53:12.716401692Z I1008 11:53:12.716114       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:53:22.716530185Z I1008 11:53:22.716343       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:53:32.716520970Z I1008 11:53:32.716340       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:53:42.717198823Z I1008 11:53:42.717073       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:53:52.717020039Z I1008 11:53:52.716842       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:54:02.716680405Z I1008 11:54:02.716505       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:54:12.716483957Z I1008 11:54:12.716298       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:54:22.716868441Z I1008 11:54:22.716688       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:54:32.717026409Z I1008 11:54:32.716838       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:54:42.716898872Z I1008 11:54:42.716729       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:54:52.716967683Z I1008 11:54:52.716805       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:55:02.716813765Z I1008 11:55:02.716614       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:55:12.717151581Z I1008 11:55:12.716994       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:55:22.716276702Z I1008 11:55:22.716104       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:55:32.716372895Z I1008 11:55:32.716206       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:55:42.717205818Z I1008 11:55:42.717036       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:55:52.716955271Z I1008 11:55:52.716772       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:56:02.716483247Z I1008 11:56:02.716304       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:56:12.716659509Z I1008 11:56:12.716486       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:56:22.717096651Z I1008 11:56:22.716936       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:56:32.716399054Z I1008 11:56:32.716245       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:56:42.716580727Z I1008 11:56:42.716421       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:56:52.717020063Z I1008 11:56:52.716841       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:57:02.716927742Z I1008 11:57:02.716768       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:57:12.717077565Z I1008 11:57:12.716931       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:57:22.717365807Z I1008 11:57:22.717190       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:57:32.716268160Z I1008 11:57:32.716098       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:57:42.716805350Z I1008 11:57:42.716541       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:57:52.716522531Z I1008 11:57:52.716386       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:58:02.717046315Z I1008 11:58:02.716884       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:58:12.716662768Z I1008 11:58:12.716445       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:58:22.717159301Z I1008 11:58:22.717011       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:58:32.716316802Z I1008 11:58:32.716127       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:58:42.716312486Z I1008 11:58:42.716155       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:58:52.716945916Z I1008 11:58:52.716781       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:59:02.716519047Z I1008 11:59:02.716363       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:59:12.716721439Z I1008 11:59:12.716549       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:59:22.716988532Z I1008 11:59:22.716825       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:59:32.716954450Z I1008 11:59:32.716795       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:59:42.716938121Z I1008 11:59:42.716797       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T11:59:52.716346703Z I1008 11:59:52.716174       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:00:02.716544075Z I1008 12:00:02.716379       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:00:12.716585204Z I1008 12:00:12.716409       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:00:22.717208255Z I1008 12:00:22.716953       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:00:32.717258521Z I1008 12:00:32.717082       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:00:42.716737817Z I1008 12:00:42.716554       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:00:52.716333931Z I1008 12:00:52.716190       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:01:02.716468985Z I1008 12:01:02.716311       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:01:12.716481241Z I1008 12:01:12.716307       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:01:22.717228013Z I1008 12:01:22.717051       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:01:32.716568548Z I1008 12:01:32.716379       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:01:42.716792911Z I1008 12:01:42.716580       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:01:52.716301802Z I1008 12:01:52.716105       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:02:02.716790099Z I1008 12:02:02.716602       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:02:12.716311233Z I1008 12:02:12.716157       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:02:22.716559030Z I1008 12:02:22.716360       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:02:32.717272560Z I1008 12:02:32.717041       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:02:42.716948824Z I1008 12:02:42.716776       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:02:52.716287546Z I1008 12:02:52.716097       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:03:02.716992470Z I1008 12:03:02.716814       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:03:12.717125512Z I1008 12:03:12.716932       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:03:22.717012692Z I1008 12:03:22.716857       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:03:32.716989630Z I1008 12:03:32.716809       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:03:42.717258917Z I1008 12:03:42.717103       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:03:52.717210218Z I1008 12:03:52.717031       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-7db6667bfc-7g46h/liveness-probe] 2024-10-08T12:04:02.716507221Z I1008 12:04:02.716341       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"

kube-system    hcloud-csi-controller-7db6667bfc-7g46h                 1/5     CrashLoopBackOff   47 (24s ago)   36m

Will be grateful for any ideas on how to move forward from this. Thank you

@lukasmetzner
Copy link
Contributor

Hey,

could you please provide some more information by:

  • Running kubectl -n kube-system describe pods hcloud-csi-controller-XXXXXXXXX-XXXXX
  • Adding the --previous flag to the log command

You could also enable debug logs via the environment variables: LOG_LEVEL=trace and HCLOUD_DEBUG=1. This might give us some detailed information.

In general the issue is related to the csi-driver container not starting correctly. As a result the unix socket is not created and the liveness probe fails and other related containers are not working correctly.

Best Regards
Lukas

@sloniki
Copy link

sloniki commented Oct 20, 2024

Having the same issue.
More details:

root@master:~# kubectl version
Client Version: v1.30.5+k3s1
Server Version: v1.30.5+k3s1
root@master:~# kubectl -n kube-system describe pods hcloud-csi-controller-6c5f68995c-j7sng
Name:             hcloud-csi-controller-6c5f68995c-j7sng
Namespace:        kube-system
Priority:         0
Service Account:  hcloud-csi-controller
Node:             worker/10.0.0.2
Start Time:       Sun, 20 Oct 2024 16:32:01 +0000
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=hcloud-csi
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=hcloud-csi
                  helm.sh/chart=hcloud-csi-2.9.0
                  pod-template-hash=6c5f68995c
Annotations:      <none>
Status:           Running
IP:               10.42.1.2
IPs:
  IP:           10.42.1.2
Controlled By:  ReplicaSet/hcloud-csi-controller-6c5f68995c
Containers:
  csi-attacher:
    Container ID:  containerd://c44294e8bfebbfc7ee14a92110bf18485b090cdd068a50a6cd4c86bf38a72c0d
    Image:         registry.k8s.io/sig-storage/csi-attacher:v4.6.1
    Image ID:      registry.k8s.io/sig-storage/csi-attacher@sha256:b4d611100ece2f9bc980d1cb19c2285b8868da261e3b1ee8f45448ab5512ab94
    Port:          <none>
    Host Port:     <none>
    Args:
      --default-fstype=ext4
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 20 Oct 2024 16:46:42 +0000
      Finished:     Sun, 20 Oct 2024 16:47:12 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 20 Oct 2024 16:41:07 +0000
      Finished:     Sun, 20 Oct 2024 16:41:37 +0000
    Ready:          False
    Restart Count:  7
    Environment:    <none>
    Mounts:
      /run/csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c9gcj (ro)
  csi-resizer:
    Container ID:   containerd://16fec330133ad1d6b612b2a9c2f741a3baa07f4ad1b254271ba0dfbf4699f1b0
    Image:          registry.k8s.io/sig-storage/csi-resizer:v1.11.2
    Image ID:       registry.k8s.io/sig-storage/csi-resizer@sha256:be6a7de1d43dba90710b61bd3d0d8f568654a6adadaeea9188cf4cd3554cbb87
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 20 Oct 2024 16:46:42 +0000
      Finished:     Sun, 20 Oct 2024 16:47:12 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 20 Oct 2024 16:41:07 +0000
      Finished:     Sun, 20 Oct 2024 16:41:37 +0000
    Ready:          False
    Restart Count:  7
    Environment:    <none>
    Mounts:
      /run/csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c9gcj (ro)
  csi-provisioner:
    Container ID:  containerd://98326f897e0e1cf84278b22b92485c85a6895e70f3ec52b9f7180bbb2ed16dec
    Image:         registry.k8s.io/sig-storage/csi-provisioner:v5.0.2
    Image ID:      registry.k8s.io/sig-storage/csi-provisioner@sha256:7b9cdb5830d01bda96111b4f138dbddcc01eed2f95aa980a404c45a042d60a10
    Port:          <none>
    Host Port:     <none>
    Args:
      --feature-gates=Topology=true
      --default-fstype=ext4
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 20 Oct 2024 16:46:42 +0000
      Finished:     Sun, 20 Oct 2024 16:47:12 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 20 Oct 2024 16:41:07 +0000
      Finished:     Sun, 20 Oct 2024 16:41:37 +0000
    Ready:          False
    Restart Count:  7
    Environment:    <none>
    Mounts:
      /run/csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c9gcj (ro)
  liveness-probe:
    Container ID:   containerd://9c9ab32e63a5c68eaa595f426ecc2b1746bbec17da8870fcf0173180fe7ca350
    Image:          registry.k8s.io/sig-storage/livenessprobe:v2.13.1
    Image ID:       registry.k8s.io/sig-storage/livenessprobe@sha256:d2a9027a4876e039185e9bef7c61a0142c8ea14e7440860285c34ac73fee4ffb
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sun, 20 Oct 2024 16:32:10 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /run/csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c9gcj (ro)
  hcloud-csi-driver:
    Container ID:  containerd://71e21ce73ed4feee64f51565a8ecff073c8cce4a448336376e9b6a4f917bbb04
    Image:         docker.io/hetznercloud/hcloud-csi-driver:v2.9.0
    Image ID:      docker.io/hetznercloud/hcloud-csi-driver@sha256:194472f0f80b67563e988b23dc7b0be2d41f09a7e0789cd0900f67365995885a
    Port:          9808/TCP
    Host Port:     0/TCP
    Command:
      /bin/hcloud-csi-driver-controller
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Sun, 20 Oct 2024 16:44:47 +0000
      Finished:     Sun, 20 Oct 2024 16:45:05 +0000
    Ready:          False
    Restart Count:  9
    Liveness:       http-get http://:healthz/healthz delay=10s timeout=3s period=2s #success=1 #failure=5
    Environment:
      CSI_ENDPOINT:    unix:///run/csi/socket
      ENABLE_METRICS:  false
      KUBE_NODE_NAME:   (v1:spec.nodeName)
      HCLOUD_TOKEN:    <set to the key 'token' in secret 'hcloud'>  Optional: false
    Mounts:
      /run/csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c9gcj (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  socket-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-c9gcj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  15m                   default-scheduler  Successfully assigned kube-system/hcloud-csi-controller-6c5f68995c-j7sng to worker
  Normal   Pulling    15m                   kubelet            Pulling image "registry.k8s.io/sig-storage/csi-attacher:v4.6.1"
  Normal   Pulled     15m                   kubelet            Successfully pulled image "registry.k8s.io/sig-storage/csi-attacher:v4.6.1" in 2.186s (2.186s including waiting). Image size: 27134440 bytes.
  Normal   Created    15m                   kubelet            Created container csi-attacher
  Normal   Started    15m                   kubelet            Started container csi-attacher
  Normal   Pulling    15m                   kubelet            Pulling image "registry.k8s.io/sig-storage/csi-resizer:v1.11.2"
  Normal   Pulled     15m                   kubelet            Successfully pulled image "registry.k8s.io/sig-storage/csi-resizer:v1.11.2" in 2.185s (2.185s including waiting). Image size: 27301963 bytes.
  Normal   Created    15m                   kubelet            Created container csi-resizer
  Normal   Started    15m                   kubelet            Started container csi-resizer
  Normal   Pulling    15m                   kubelet            Pulling image "registry.k8s.io/sig-storage/csi-provisioner:v5.0.2"
  Normal   Pulled     15m                   kubelet            Successfully pulled image "registry.k8s.io/sig-storage/csi-provisioner:v5.0.2" in 2.087s (2.087s including waiting). Image size: 28422483 bytes.
  Normal   Created    15m                   kubelet            Created container csi-provisioner
  Normal   Started    15m                   kubelet            Started container csi-provisioner
  Normal   Pulled     15m                   kubelet            Container image "registry.k8s.io/sig-storage/livenessprobe:v2.13.1" already present on machine
  Normal   Created    15m                   kubelet            Created container liveness-probe
  Normal   Started    15m                   kubelet            Started container liveness-probe
  Normal   Created    15m                   kubelet            Created container hcloud-csi-driver
  Normal   Started    15m                   kubelet            Started container hcloud-csi-driver
  Normal   Pulled     14m (x2 over 15m)     kubelet            Container image "docker.io/hetznercloud/hcloud-csi-driver:v2.9.0" already present on machine
  Warning  Unhealthy  14m (x5 over 15m)     kubelet            Liveness probe failed: Get "http://10.42.1.2:9808/healthz": dial tcp 10.42.1.2:9808: connect: connection refused
  Normal   Killing    14m                   kubelet            Container hcloud-csi-driver failed liveness probe, will be restarted
  Warning  BackOff    5m17s (x39 over 14m)  kubelet            Back-off restarting failed container csi-attacher in pod hcloud-csi-controller-6c5f68995c-j7sng_kube-system(37b0a058-eecc-410a-abe1-fa7e87ea8982)
  Warning  BackOff    12s (x66 over 13m)    kubelet            Back-off restarting failed container hcloud-csi-driver in pod hcloud-csi-controller-6c5f68995c-j7sng_kube-system(37b0a058-eecc-410a-abe1-fa7e87ea8982)

@lerodu
Copy link

lerodu commented Nov 7, 2024

I'm having the same problem.

root@control-plane-0:~# kubectl version
Client Version: v1.30.6+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.6+k3s1
root@control-plane-0:~# kubectl describe pod hcloud-csi-controller-5b779974d8-22vp2 -n kube-system
Name:             hcloud-csi-controller-5b779974d8-22vp2
Namespace:        kube-system
Priority:         0
Service Account:  hcloud-csi-controller
Node:             control-plane-1/10.0.1.2
Start Time:       Thu, 07 Nov 2024 16:30:13 +0000
Labels:           app=hcloud-csi-controller
                  app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=hcloud-csi
                  app.kubernetes.io/name=hcloud-csi
                  pod-template-hash=5b779974d8
Annotations:      <none>
Status:           Running
IP:               10.42.5.19
IPs:
  IP:           10.42.5.19
Controlled By:  ReplicaSet/hcloud-csi-controller-5b779974d8
Containers:
  csi-attacher:
    Container ID:  containerd://999e6805cc5a856620d3413c84decb22eac698276d461fd3b4f73ee2e22e18ea
    Image:         registry.k8s.io/sig-storage/csi-attacher:v4.6.1
    Image ID:      registry.k8s.io/sig-storage/csi-attacher@sha256:b4d611100ece2f9bc980d1cb19c2285b8868da261e3b1ee8f45448ab5512ab94
    Port:          <none>
    Host Port:     <none>
    Args:
      --default-fstype=ext4
    State:          Running
      Started:      Thu, 07 Nov 2024 16:55:39 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 07 Nov 2024 16:49:58 +0000
      Finished:     Thu, 07 Nov 2024 16:50:28 +0000
    Ready:          True
    Restart Count:  9
    Environment:    <none>
    Mounts:
      /run/csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5j4xk (ro)
  csi-resizer:
    Container ID:   containerd://89adfc6381268ca0d567aa18d492e1a6a21d90b0368d85f39964c1d748dd2b86
    Image:          registry.k8s.io/sig-storage/csi-resizer:v1.11.2
    Image ID:       registry.k8s.io/sig-storage/csi-resizer@sha256:be6a7de1d43dba90710b61bd3d0d8f568654a6adadaeea9188cf4cd3554cbb87
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 07 Nov 2024 16:55:39 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 07 Nov 2024 16:49:58 +0000
      Finished:     Thu, 07 Nov 2024 16:50:28 +0000
    Ready:          True
    Restart Count:  9
    Environment:    <none>
    Mounts:
      /run/csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5j4xk (ro)
  csi-provisioner:
    Container ID:  containerd://5b3b48bc8a341086f8622412f4c62e07ed3d94038725f6b68b75e644c29fea10
    Image:         registry.k8s.io/sig-storage/csi-provisioner:v5.0.2
    Image ID:      registry.k8s.io/sig-storage/csi-provisioner@sha256:7b9cdb5830d01bda96111b4f138dbddcc01eed2f95aa980a404c45a042d60a10
    Port:          <none>
    Host Port:     <none>
    Args:
      --feature-gates=Topology=true
      --default-fstype=ext4
    State:          Running
      Started:      Thu, 07 Nov 2024 16:55:39 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 07 Nov 2024 16:49:58 +0000
      Finished:     Thu, 07 Nov 2024 16:50:28 +0000
    Ready:          True
    Restart Count:  9
    Environment:    <none>
    Mounts:
      /run/csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5j4xk (ro)
  liveness-probe:
    Container ID:   containerd://47c61db9c01a4fe8f1278673d5f39b6b0fe15bf6100fec7106f2607c8c476455
    Image:          registry.k8s.io/sig-storage/livenessprobe:v2.13.1
    Image ID:       registry.k8s.io/sig-storage/livenessprobe@sha256:d2a9027a4876e039185e9bef7c61a0142c8ea14e7440860285c34ac73fee4ffb
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 07 Nov 2024 16:30:14 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /run/csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5j4xk (ro)
  hcloud-csi-driver:
    Container ID:  containerd://278182bb15276e3172e5c5978c5655f1340a106ac44fe1cfa8b00e730e221ff8
    Image:         docker.io/hetznercloud/hcloud-csi-driver:v2.9.0
    Image ID:      docker.io/hetznercloud/hcloud-csi-driver@sha256:194472f0f80b67563e988b23dc7b0be2d41f09a7e0789cd0900f67365995885a
    Ports:         9189/TCP, 9808/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /bin/hcloud-csi-driver-controller
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Thu, 07 Nov 2024 16:54:06 +0000
      Finished:     Thu, 07 Nov 2024 16:54:24 +0000
    Ready:          False
    Restart Count:  13
    Liveness:       http-get http://:healthz/healthz delay=10s timeout=3s period=2s #success=1 #failure=5
    Environment:
      CSI_ENDPOINT:      unix:///run/csi/socket
      METRICS_ENDPOINT:  0.0.0.0:9189
      ENABLE_METRICS:    true
      KUBE_NODE_NAME:     (v1:spec.nodeName)
      HCLOUD_TOKEN:      <set to the key 'token' in secret 'hcloud'>  Optional: false
    Mounts:
      /run/csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5j4xk (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  socket-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-5j4xk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  25m                   default-scheduler  Successfully assigned kube-system/hcloud-csi-controller-5b779974d8-22vp2 to control-plane-1
  Normal   Pulled     25m                   kubelet            Container image "registry.k8s.io/sig-storage/livenessprobe:v2.13.1" already present on machine
  Normal   Created    25m                   kubelet            Created container csi-attacher
  Normal   Started    25m                   kubelet            Started container csi-provisioner
  Normal   Pulled     25m                   kubelet            Container image "registry.k8s.io/sig-storage/csi-resizer:v1.11.2" already present on machine
  Normal   Created    25m                   kubelet            Created container csi-resizer
  Normal   Started    25m                   kubelet            Started container csi-resizer
  Normal   Pulled     25m                   kubelet            Container image "registry.k8s.io/sig-storage/csi-provisioner:v5.0.2" already present on machine
  Normal   Created    25m                   kubelet            Created container csi-provisioner
  Normal   Started    25m                   kubelet            Started container csi-attacher
  Normal   Pulled     25m                   kubelet            Container image "registry.k8s.io/sig-storage/csi-attacher:v4.6.1" already present on machine
  Normal   Started    25m                   kubelet            Started container liveness-probe
  Normal   Created    25m                   kubelet            Created container liveness-probe
  Normal   Created    25m (x2 over 25m)     kubelet            Created container hcloud-csi-driver
  Normal   Pulled     25m (x2 over 25m)     kubelet            Container image "docker.io/hetznercloud/hcloud-csi-driver:v2.9.0" already present on machine
  Normal   Started    25m (x2 over 25m)     kubelet            Started container hcloud-csi-driver
  Normal   Killing    25m                   kubelet            Container hcloud-csi-driver failed liveness probe, will be restarted
  Warning  Unhealthy  25m (x6 over 25m)     kubelet            Liveness probe failed: Get "http://10.42.5.19:9808/healthz": dial tcp 10.42.5.19:9808: connect: connection refused
  Warning  BackOff    5m34s (x93 over 24m)  kubelet            Back-off restarting failed container hcloud-csi-driver in pod hcloud-csi-controller-5b779974d8-22vp2_kube-system(7db96885-5175-4748-9db7-5b308ed59197)
  Warning  BackOff    45s (x116 over 24m)   kubelet            Back-off restarting failed container csi-attacher in pod hcloud-csi-controller-5b779974d8-22vp2_kube-system(7db96885-5175-4748-9db7-5b308ed59197)
root@control-plane-0:~# kubectl logs hcloud-csi-controller-5b779974d8-p7khx  -n kube-system --previous --all-containers --prefix --timestamps
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-attacher] 2024-11-07T17:15:18.749192514Z I1107 17:15:18.746914       1 main.go:109] "Version" version="v4.6.1"
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-attacher] 2024-11-07T17:15:28.749553178Z I1107 17:15:28.749314       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-attacher] 2024-11-07T17:15:38.748610402Z I1107 17:15:38.748427       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-attacher] 2024-11-07T17:15:48.748606480Z I1107 17:15:48.748475       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-attacher] 2024-11-07T17:15:48.748845602Z E1107 17:15:48.748781       1 main.go:149] "Failed to connect to the CSI driver" err="context deadline exceeded" csiAddress="/run/csi/socket"
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-resizer] 2024-11-07T17:15:18.800541527Z I1107 17:15:18.800338       1 main.go:108] "Version" version="v1.11.2"
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-resizer] 2024-11-07T17:15:28.802229678Z I1107 17:15:28.802010       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-resizer] 2024-11-07T17:15:38.802231387Z I1107 17:15:38.802069       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-resizer] 2024-11-07T17:15:48.802461145Z I1107 17:15:48.802331       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-resizer] 2024-11-07T17:15:48.802698187Z E1107 17:15:48.802631       1 main.go:153] "Failed to create CSI client" err="failed to connect to CSI driver: context deadline exceeded"
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-provisioner] 2024-11-07T17:15:18.869105946Z W1107 17:15:18.868868       1 feature_gate.go:246] Setting GA feature gate Topology=true. It will be removed in a future release.
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-provisioner] 2024-11-07T17:15:18.869164346Z I1107 17:15:18.868977       1 csi-provisioner.go:154] Version: v5.0.2
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-provisioner] 2024-11-07T17:15:18.869170307Z I1107 17:15:18.868984       1 csi-provisioner.go:177] Building kube configs for running in cluster...
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-provisioner] 2024-11-07T17:15:28.870924897Z I1107 17:15:28.870610       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-provisioner] 2024-11-07T17:15:38.871202646Z I1107 17:15:38.870898       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-provisioner] 2024-11-07T17:15:48.870103838Z I1107 17:15:48.869852       1 connection.go:253] "Still connecting" address="unix:///run/csi/socket"
[pod/hcloud-csi-controller-5b779974d8-p7khx/csi-provisioner] 2024-11-07T17:15:48.870134838Z E1107 17:15:48.869970       1 csi-provisioner.go:215] context deadline exceeded
Error from server (BadRequest): previous terminated container "liveness-probe" in pod "hcloud-csi-controller-5b779974d8-p7khx" not found

@sloniki
Copy link

sloniki commented Nov 13, 2024

What helped me is providing the right kubelet args and having hccm properly installed.

  1. Install k3s with these parameters:
    curl -sfL https://get.k3s.io | sh -s - --kubelet-arg="cloud-provider=external" —disable-cloud-controller

  2. Create hcloud secret and install hccm
    helm install hccm hcloud/hcloud-cloud-controller-manager -n kube-system

  3. Install csi-driver
    helm install hcloud-csi hcloud/hcloud-csi -n kube-system

# kubectl get pods -n kube-system
NAME                                               READY   STATUS      RESTARTS   AGE
coredns-7b98449c4-7ljnd                            1/1     Running     0          10m
hcloud-cloud-controller-manager-6df5f97677-4d9hh   1/1     Running     0          7m13s
hcloud-csi-controller-b47c856c7-6d8cd              5/5     Running     0          6m17s
hcloud-csi-node-5s8qv                              3/3     Running     0          6m17s
helm-install-traefik-crd-j5mzc                     0/1     Completed   0          10m
helm-install-traefik-mqg5p                         0/1     Completed   1          10m
local-path-provisioner-595dcfc56f-c7zvc            1/1     Running     0          10m
metrics-server-cdcc87586-bh2k6                     1/1     Running     0          10m
svclb-traefik-7b0764b7-wtg2k                       2/2     Running     0          6m57s
traefik-d7c9c5778-97br4                            1/1     Running     0          6m57s

@lukasmetzner
Copy link
Contributor

Hey,

@wanGiB @lerodu @sloniki could you provide detailed steps to reproduce your issue?

Best Regards
Lukas

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

4 participants