Mount failed with error: rpc error: [failed to authenticate credentials for azstorage]

[Hidayathullashaik]

Hi, I am getting a repitative error for azure blob credentials while provisioning the PV, PVCs in the Kubernetes [K3s] Cluster using blobfuse protocol. I have verified the az blob storage account credentials working in other clusters. Need assistance/support to fix the issue.

Error:
MountVolume.MountDevice failed for volume "industrial-pv-blob" : rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.

CSI Blob Node Logs
volumeId _pepsensecst-demo
context map[containerName:pepsensecst-demo protocol:fuse2 resourceGroup: storageAccount:]
mountflags [-o allow_other --file-cache-timeout-in-seconds=120]
mountOptions [-o allow_other --file-cache-timeout-in-seconds=120 --empty-dir-check=false --tmp-path=/mnt/-demo --container-name=pepsensecst-demo --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10]
args /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/ac57e1b1e08147bd8749693401e850fc7d31875873642a7150780425d14de8bc/globalmount -o allow_other --file-cache-timeout-in-seconds=120 --empty-dir-check=false --tmp-path=/mnt/_pepsensecst-demo --container-name=pepsensecst-demo --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10
serverAddress .blob.core.windows.net
I1111 12:05:04.424943 1244193 nodeserver.go:154] start connecting to blobfuse proxy, protocol: fuse2, args: /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/ac57e1b1e08147bd8749693401e850fc7d31875873642a7150780425d14de8bc/globalmount -o allow_other --file-cache-timeout-in-seconds=120 --empty-dir-check=false --tmp-path=/mnt/_pepsensecst-demo --container-name=pepsensecst-demo --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10
I1111 12:05:04.430847 1244193 nodeserver.go:163] begin to mount with blobfuse proxy, protocol: fuse2, args: /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/ac57e1b1e08147bd8749693401e850fc7d31875873642a7150780425d14de8bc/globalmount -o allow_other --file-cache-timeout-in-seconds=120 --empty-dir-check=false --tmp-path=/mnt/_pepsensecst-demo --container-name=pepsensecst-demo --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10
E1111 12:05:06.910818 1244193 nodeserver.go:166] GRPC call returned with an error:rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
E1111 12:05:06.910971 1244193 nodeserver.go:392] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
, output:
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.
E1111 12:05:06.911130 1244193 utils.go:80] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
, output:
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.
I1111 12:05:07.446630 1244193 utils.go:75] GRPC call: /csi.v1.Node/NodeStageVolume
I1111 12:05:07.446664 1244193 utils.go:76] GRPC request: {"secrets":"stripped","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/ac57e1b1e08147bd8749693401e850fc7d31875873642a7150780425d14de8bc/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120"]}},"access_mode":{"mode":5}},"volume_context":{"containerName":"pepsensecst-demo","protocol":"fuse2","resourceGroup":"","storageAccount":""},"volume_id":"_pepsensecst-demo"}
I1111 12:05:07.447554 1244193 blob.go:386] parsing volumeID(_pepsensecst-demo) return with error: error parsing volume id: "_pepsensecst-demo", should at least contain two #
I1111 12:05:07.447591 1244193 blob.go:458] volumeID(_pepsensecst-demo) authEnv: []
I1111 12:05:07.447654 1244193 nodeserver.go:366] target /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/ac57e1b1e08147bd8749693401e850fc7d31875873642a7150780425d14de8bc/globalmountprotocol fuse2

Validations
I have validated the connectivity from the vm to storage account and shared the details below
[root@peplapsete01 ~]#
[root@peplapsete01 ~]# nc -zv .blob.core.windows.net 443
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 20.209.146.193:443.
Ncat: 0 bytes sent, 0 bytes received in 0.57 seconds.
[root@peplapsete01 ~]#
[root@peplapsete01 ~]#

Configurations
PV.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: blob.csi.azure.com
name: industrial-pv-blob
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: blob-fuse
mountOptions:
- -o allow_other
- --file-cache-timeout-in-seconds=120
csi:
driver: blob.csi.azure.com
readOnly: false
volumeHandle: _pepsensecst-demo
volumeAttributes:
resourceGroup:
storageAccount:
containerName: pepsensecst-demo
protocol: fuse2
nodeStageSecretRef:
name: industrialqa-secret
namespace: default

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: industrial-pvc-blob
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
volumeName: industrial-pv-blob
storageClassName: blob-fuse

kind: Pod
apiVersion: v1
metadata:
name: nginx-blob
spec:
nodeSelector:
"kubernetes.io/os": linux
containers:
- image: mcr.microsoft.com/oss/nginx/nginx:1.17.3-alpine
name: nginx-blob
command:
- "/bin/sh"
- "-c"
- while true; do echo $(date) >> /mnt/blob/outfile; sleep 1; done
volumeMounts:
- name: blob01
mountPath: "/mnt/blob"
resources:
limits:
cpu: 0.5
memory: "512Mi"
requests:
cpu: 0.1
memory: "128Mi"
volumes:
- name: blob01
persistentVolumeClaim:
claimName: industrial-pvc-blob

Expected Behaviour
Blobfuse should make connection from the vm to the azurestorageaccount and provision a volume successfully in the container.

[andyzhangx]

@Hidayathullashaik there is auth error, how did you create the industrialqa-secret?

you could use following way to create the secret with accoutname and key:

kubectl create secret generic industrialqa-secret --from-literal=azurestorageaccountname="xxx" --from-literal azurestorageaccountkey="xxx" --type=Opaque

[Hidayathullashaik]

@andyzhangx - Hi, Thank you for the suggestion. I have already created the industrialqa-secret before creating the PV, PVCs in both the ways using the Rancher UI Portal & through the imperative kubectl commands with the preveleged credentials but I'm still encountering the same error:

MountVolume.MountDevice failed for volume "industrial-pv-blob" : rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.

Please let me know if there are any additional steps or configurations I should try.

[andyzhangx]

@Hidayathullashaik to narrow down the issue, you could follow this guide to check whether blobfuse mount works on the agent node: https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/csi-debug.md#troubleshooting-connection-failure-on-agent-node

[Hidayathullashaik]

Hi @andyzhangx,

After re-installing K3s, I am now encountering a new error:

Error:
MountVolume.MountDevice failed for volume "industrial-pv-blob" : rpc error: code = Internal desc = Mount failed with error: context deadline exceeded, output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.

This is a different error compared to the previous one. Could you please provide guidance on how to resolve this new issue? Any insights would be greatly appreciated.

[andyzhangx]

@Hidayathullashaik pls follow this guide to check whether blobfuse mount works on the agent node: https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/csi-debug.md#troubleshooting-connection-failure-on-agent-node

[Hidayathullashaik]

@andyzhangx - Getting the same authentication failed error for the blobfuse2 mount in all the nodes. I have exported the Storage account name and key as well.

[root@peplapsete01 ~]#
[root@peplapsete01 ~]# blobfuse2 test --container-name=nginx-blob --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120
Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
[root@peplapsete01 ~]#
[root@peplapsete01 ~]#
#######
[pepsense@peplapsete02 ~]$
[pepsense@peplapsete02 ~]$ blobfuse2 test --container-name=nginx-blob --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120
Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
[pepsense@peplapsete02 ~]$
[pepsense@peplapsete02 ~]$

[pepsense@peplapsete03 ~]$
[pepsense@peplapsete03 ~]$ blobfuse2 test --container-name=nginx-blob --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120
Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
[pepsense@peplapsete03 ~]$

[andyzhangx]

pls check the /var/log/blobfuse2.log to get the detailed error msg

[Hidayathullashaik]

@andyzhangx - I’ve reviewed the logs in /var/log/blobfuse2.log, and here are the relevant error details I found:

Nov 14 03:57:20 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_CRIT [mount.go (432)]: Starting Blobfuse2 Mount : 2.3.2 on [Red Hat Enterprise Linux 8.8 (Ootpa)]
Nov 14 03:57:20 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_CRIT [mount.go (434)]: Logging level set to : LOG_WARNING
Nov 14 03:57:20 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [file_cache.go (252)]: FileCache: config error [tmp-path does not exist. attempting to create tmp-path.]
Nov 14 03:57:20 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_WARNING [config.go (362)]: ParseAndValidateConfig : account endpoint not provided, assuming the default .core.windows.net style endpoint
Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [block_blob.go (199)]: BlockBlob::TestPipeline : Failed to validate account with given auth %!s(func() string=0x88cb80)
Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [azstorage.go (161)]: AzStorage::configureAndTest : Failed to validate credentials [GET https://xxxx.blob.core.windows.net/nginx-blob#012--------------------------------------------------------------------------------#012RESPONSE 403: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.#012ERROR CODE: AuthenticationFailed#012--------------------------------------------------------------------------------#12AuthenticationFailedServer failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.#012RequestId:71f252d3-b01e-002c-0b7c-366052000000#012Time:2024-11-14T10:04:16.3103585ZRequest date header too old: 'Thu, 14 Nov 2024 06:57:20 GMT'#12--------------------------------------------------------------------------------#12]
Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [azstorage.go (101)]: AzStorage::Configure : Failed to validate storage account [failed to authenticate credentials for azstorage]
Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [pipeline.go (69)]: Pipeline: error creating pipeline component azstorage [failed to authenticate credentials for azstorage]
Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [mount.go (442)]: mount : failed to initialize new pipeline [failed to authenticate credentials for azstorage]
Nov 14 03:58:00 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_CRIT [mount.go (432)]: Starting Blobfuse2 Mount : 2.3.2 on [Red Hat Enterprise Linux 8.8 (Ootpa)]
Nov 14 03:58:00 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_CRIT [mount.go (434)]: Logging level set to : LOG_WARNING
Nov 14 03:58:00 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_WARNING [config.go (362)]: ParseAndValidateConfig : account endpoint not provided, assuming the default .core.windows.net style endpoint
Nov 14 03:58:01 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_ERR [block_blob.go (199)]: BlockBlob::TestPipeline : Failed to validate account with given auth %!s(func() string=0x88cb80)
Nov 14 03:58:01 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_ERR [azstorage.go (161)]: AzStorage::configureAndTest : Failed to validate credentials [GET https://xxxx.blob.core.windows.net/nginx-blob#012--------------------------------------------------------------------------------#012RESPONSE 403: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.#012ERROR CODE: AuthenticationFailed#012--------------------------------------------------------------------------------#12AuthenticationFailedServer failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.#012RequestId:a5ec1e1b-201e-0091-647c-36f43c000000#012Time:2024-11-14T10:04:56.1537364ZRequest date header too old: 'Thu, 14 Nov 2024 06:58:00 GMT'#12--------------------------------------------------------------------------------#12]
Nov 14 03:58:01 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_ERR [azstorage.go (101)]: AzStorage::Configure : Failed to validate storage account [failed to authenticate credentials for azstorage]
Nov 14 03:58:01 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_ERR [pipeline.go (69)]: Pipeline: error creating pipeline component azstorage [failed to authenticate credentials for azstorage]
Nov 14 03:58:01 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_ERR [mount.go (442)]: mount : failed to initialize new pipeline [failed to authenticate cre:

[andyzhangx]

have you provided the right account key? you could file issue here: https://github.com/Azure/azure-storage-fuse/issues since it's a blobfuse auth issue.

[Hidayathullashaik]

@andyzhangx - Yes, I have reviewed the storage account name & key and it is correct. Sure will log the issue in the above shared url. Thankyou for your support.