Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mount failed with error: rpc error: [failed to authenticate credentials for azstorage] #1566

Open
Hidayathullashaik opened this issue Nov 14, 2024 · 11 comments
Open

Mount failed with error: rpc error: [failed to authenticate credentials for azstorage] #1566

Hidayathullashaik opened this issue Nov 14, 2024 · 11 comments
Assignees
@vibhansa-msft
Labels
V2
Milestone
v2-2.4.0

Comments

@Hidayathullashaik
Copy link

Hi, I am getting authentication error for azure blob credentials while provisioning the PV, PVCs in the Kubernetes [K3s] Cluster using blobfuse protocol. I have verified the az blob storage account credentials working in other clusters. Need assistance/support to fix the issue.

Error:
MountVolume.MountDevice failed for volume "industrial-pv-blob" : rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.

Getting the same authentication failed error with the blobfuse2 commands mount in all the nodes. I have exported the Storage account name and key as well.

[root@peplapsete01 ~]#
[root@peplapsete01 ~]# blobfuse2 test --container-name=nginx-blob --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120
Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
[root@peplapsete01 ~]#
[root@peplapsete01 ~]#
#######
[pepsense@peplapsete02 ~]$
[pepsense@peplapsete02 ~]$ blobfuse2 test --container-name=nginx-blob --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120
Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
[pepsense@peplapsete02 ~]$
[pepsense@peplapsete02 ~]$

[pepsense@peplapsete03 ~]$
[pepsense@peplapsete03 ~]$ blobfuse2 test --container-name=nginx-blob --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120
Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
[pepsense@peplapsete03 ~]$

the logs in /var/log/blobfuse2.log:

Nov 14 03:57:20 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_CRIT [mount.go (432)]: Starting Blobfuse2 Mount : 2.3.2 on [Red Hat Enterprise Linux 8.8 (Ootpa)]
Nov 14 03:57:20 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_CRIT [mount.go (434)]: Logging level set to : LOG_WARNING
Nov 14 03:57:20 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [file_cache.go (252)]: FileCache: config error [tmp-path does not exist. attempting to create tmp-path.]
Nov 14 03:57:20 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_WARNING [config.go (362)]: ParseAndValidateConfig : account endpoint not provided, assuming the default .core.windows.net style endpoint
Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [block_blob.go (199)]: BlockBlob::TestPipeline : Failed to validate account with given auth %!s(func() string=0x88cb80)
Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [azstorage.go (161)]: AzStorage::configureAndTest : Failed to validate credentials [GET https://xxxx.blob.core.windows.net/nginx-blob#012--------------------------------------------------------------------------------#012RESPONSE 403: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.#012ERROR CODE: AuthenticationFailed#012--------------------------------------------------------------------------------https://github.com/kubernetes-sigs/blob-csi-driver/issues/12AuthenticationFailedServer failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.#012RequestId:71f252d3-b01e-002c-0b7c-366052000000#012Time:2024-11-14T10:04:16.3103585ZRequest date header too old: 'Thu, 14 Nov 2024 06:57:20 GMT'https://github.com/kubernetes-sigs/blob-csi-driver/issues/12--------------------------------------------------------------------------------https://github.com/kubernetes-sigs/blob-csi-driver/issues/12]
Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [azstorage.go (101)]: AzStorage::Configure : Failed to validate storage account [failed to authenticate credentials for azstorage]
Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [pipeline.go (69)]: Pipeline: error creating pipeline component azstorage [failed to authenticate credentials for azstorage]
Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [mount.go (442)]: mount : failed to initialize new pipeline [failed to authenticate credentials for azstorage]
Nov 14 03:58:00 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_CRIT [mount.go (432)]: Starting Blobfuse2 Mount : 2.3.2 on [Red Hat Enterprise Linux 8.8 (Ootpa)]
Nov 14 03:58:00 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_CRIT [mount.go (434)]: Logging level set to : LOG_WARNING
Nov 14 03:58:00 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_WARNING [config.go (362)]: ParseAndValidateConfig : account endpoint not provided, assuming the default .core.windows.net style endpoint
Nov 14 03:58:01 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_ERR [block_blob.go (199)]: BlockBlob::TestPipeline : Failed to validate account with given auth %!s(func() string=0x88cb80)
Nov 14 03:58:01 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_ERR [azstorage.go (161)]: AzStorage::configureAndTest : Failed to validate credentials [GET https://xxxx.blob.core.windows.net/nginx-blob#012--------------------------------------------------------------------------------#012RESPONSE 403: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.#012ERROR CODE: AuthenticationFailed#012--------------------------------------------------------------------------------https://github.com/kubernetes-sigs/blob-csi-driver/issues/12AuthenticationFailedServer failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.#012RequestId:a5ec1e1b-201e-0091-647c-36f43c000000#012Time:2024-11-14T10:04:56.1537364ZRequest date header too old: 'Thu, 14 Nov 2024 06:58:00 GMT'https://github.com/kubernetes-sigs/blob-csi-driver/issues/12--------------------------------------------------------------------------------https://github.com/kubernetes-sigs/blob-csi-driver/issues/12]
Nov 14 03:58:01 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_ERR [azstorage.go (101)]: AzStorage::Configure : Failed to validate storage account [failed to authenticate credentials for azstorage]
Nov 14 03:58:01 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_ERR [pipeline.go (69)]: Pipeline: error creating pipeline component azstorage [failed to authenticate credentials for azstorage]
Nov 14 03:58:01 peplapsete01 blobfuse2[2503060]: [/root/test] LOG_ERR [mount.go (442)]: mount : failed to initialize new pipeline [failed to authenticate cre:

Associated blobcsi github issues: kubernetes-sigs/blob-csi-driver#1689
@vibhansa-msft
Copy link
Member

vibhansa-msft commented Nov 14, 2024 via email

@andyzhangx
Copy link
Contributor

@vibhansa-msft that's a blobfuse2 mount failure on the node, could you help check the issue?

@vibhansa-msft
Copy link
Member 

Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [block_blob.go (199)]: BlockBlob::TestPipeline : Failed to validate account with given auth %!s(func() string=0x88cb80)
Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [azstorage.go (161)]: AzStorage::configureAndTest : Failed to validate credentials

It;s an authentication failure. Kindly validate your config are fine.

@vibhansa-msft vibhansa-msft added this to the v2-2.4.0 milestone Nov 15, 2024
@vibhansa-msft vibhansa-msft self-assigned this Nov 15, 2024
@Hidayathullashaik
Copy link
Author

@vibhansa-msft - Hi, I have validated both the configurations & credentials and they are valid. It is working fine in other environments without any issues. Please let me know if there are any additional steps or configurations I should try.

@vibhansa-msft
Copy link
Member

What authentication mode you are using here ?

@Hidayathullashaik
Copy link
Author

I have exported environment variables using the reference. https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/csi-debug.md#troubleshooting-connection-failure-on-agent-node

mkdir test
export AZURE_STORAGE_ACCOUNT=
export AZURE_STORAGE_ACCESS_KEY=

only for sovereign cloud

export AZURE_STORAGE_BLOB_ENDPOINT=accountname.blob.core.chinacloudapi.cn

blobfuse2 test --container-name=CONTAINER-NAME --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120

@vibhansa-msft
Copy link
Member

Exports looks corret. You might want to export the auth_type related env variable as well and set the value to "key" as you are using key based authentication here.

@andyzhangx
Copy link
Contributor

@Hidayathullashaik same account name and key works on other node and it does not work on this node? what's the diff between these nodes?

@Hidayathullashaik
Copy link
Author

@vibhansa-msft - I have exported the auth_type variable as well but still no luck. Below shared the details
[root@peplapsete01 ~]#
[root@peplapsete01 ~]# echo $AZURE_STORAGE_AUTH_TYPE
KEY
[root@peplapsete01 ~]#
[root@peplapsete01 ~]# blobfuse2 test --container-name=nginx-blob --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120
Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
[root@peplapsete01 ~]#
[root@peplapsete01 ~]#

@Hidayathullashaik
Copy link
Author

@Hidayathullashaik same account name and key works on other node and it does not work on this node? what's the diff between these nodes?

@andyzhangx - The same storage account configurations are applied in different cluster and it is working fine. Both the clusters are configured with the same OS and Networking configurations.

@vibhansa-msft
Copy link
Member 

Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [block_blob.go (199)]: BlockBlob::TestPipeline : Failed to validate account with given auth %!s(func() string=0x88cb80)
Nov 14 03:57:21 peplapsete01 blobfuse2[2502872]: [/root/test] LOG_ERR [azstorage.go (161)]: AzStorage::configureAndTest : Failed to validate credentials [GET https://xxxx.blob.core.windows.net/nginx-blob#012--------------------------------------------------------------------------------#012RESPONSE 403: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.#012ERROR CODE: AuthenticationFailed#012--------------------------------------------------------------------------------

This log clearly indicates something wrong with the authorization/authentication. Either some env variable is not exposed or some invalid format is there. Just to validate you can manually mount outside AKS with same config and check it works or not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vibhansa-msft vibhansa-msft

Labels
V2
Projects
None yet
Milestone
v2-2.4.0
Development

No branches or pull requests
3 participants
@andyzhangx @vibhansa-msft @Hidayathullashaik