node-driver-registrar v2.6.0 fails to create /var/lib/kubelet/plugins/csi-secrets-store/registration

[haugom]

What steps did you take and what happened:

1. I used the helm chart secrets-store-csi-driver to install the secrets-store-csi-driver with the v2.6.0 version of the node-driver-registrar
2. the daemonset crashes

main.go:107] "Failed to create registration probe file" err="open /var/lib/kubelet/plugins/csi-secrets-store/registration: no such file or directory" registrationProbePath="/var/lib/kubelet/plugins/csi-secrets-store/registration"

What did you expect to happen:

The daemonset should succeed.

Anything else you would like to add:

I maybe be using the helm chart wrong or have missed something. I had a bot that detected a new version of node-driver-registrar (and liveness) images and it updated my values for me.

I checked the changelog but did not see any breaking change so I rolled it out.

After reading the code and trying to locate the error message I believe the error is this:

• this PR Remove mkdir call while creating the registration probe file kubernetes-csi/node-driver-registrar#214 removed some code that will make all the parts of the directory for the --kubelet-registration-path argument to node-driver-registrar
• the helm chart (and static deployment file here: deploy/secrets-store-csi-driver.yaml both pass --kubelet-registration-path argument and set it to /var/lib/kubelet/plugins/csi-secrets-store/csi.sock
• the code picks up the directory name from that and decides to try create a file /var/lib/kubelet/plugins/csi-secrets-store/registration which failes now that the code was changed in PR 214

the reason it fails as is because the hostPath folder /var/lib/kubelet/plugins/csi-secrets-store is mounted in the containers as /csi and not /var/lib/kubelet/plugins/csi-secrets-store

The node-driver-registrar wants to create a file registration and tries to resolve the path by using the directory name of the --kubelet-registration-path and concatenating it with /registration, but that folder does not exist when the container runs because the it is mounted under /csi

Which provider are you using:

For this issue it does not matter but I am using vault-csi-driver.

Environment:

• Secrets Store CSI Driver version: v1.2.4
• node-driver-registrar: v2.6.0
• Kubernetes version: 1.23

[aramase]

@haugom We have not updated to v2.6.0 for node-driver-registrar yet. We will test and update when ready. I would recommend using only the manifests and images tagged in this repo as they're tested. For any other versions used, there is no support.

/close

[k8s-ci-robot]

@aramase: Closing this issue.

In response to this:

@haugom We have not updated to v2.6.0 for node-driver-registrar yet. We will test and update when ready. I would recommend using only the manifests and images tagged in this repo as they're tested. For any other versions used, there is no support.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sarahhenkens]

Ran into this same problem. With v2.6.1 releases as well. I think its time to update the helm charts IMO

[sarahhenkens]

Can we reopen this @aramase?

[aramase]

@sarahhenkens The helm charts work fine now with node-driver-registrar v2.5.1. We have not upgraded to the v2.6.1 release and as mentioned in the thread above, we recommend only using the versions we have in the repo which are tested. There is a bug in v2.6.0 and v2.6.1 release which is being discussed here: kubernetes-csi/node-driver-registrar#244