-
Notifications
You must be signed in to change notification settings - Fork 206
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't access file in subdir ("No such file or directory" when accessing) #866
Comments
Hi @XavierGeerinck, Thank you for reporting this issue. Could you please share if you created these directories using blobfuse? Blob storage has flat namespace so it only has the concept of "virtual" directories. Blobfuse2 gets around this by creating marker directories and setting special metadata on the directories. If these marker directories don't exist, Blobfuse can't find the intermediate folders (which cat expects to exist). This case will work seamlessly with hierarchical namespace accounts which do have the concept of directories, so all those intermediate directories will be added when uploading the file on portal |
I did not create those using blob fuse. Is there any way to add this metadata afterwards or should I use adls?
Get Outlook for iOS<https://aka.ms/o0ukef>
…________________________________
From: Gauri Prasad ***@***.***>
Sent: Wednesday, August 17, 2022 6:39:53 PM
To: Azure/azure-storage-fuse ***@***.***>
Cc: Xavier Geerinck ***@***.***>; Mention ***@***.***>
Subject: Re: [Azure/azure-storage-fuse] Can't access file in subdir ("No such file or directory" when accessing) (Issue #866)
Hi @XavierGeerinck<https://github.com/XavierGeerinck>, Thank you for reporting this issue. Could you please share if you created these directories using blobfuse? Blob storage has flat namespace so it only has the concept of "virtual" directories. Blobfuse2 gets around this by creating marker directories and setting special metadata on the directories. If these marker directories don't exist, Blobfuse can't find the intermediate folders (which cat expects to exist).
This case will work seamlessly with hierarchical namespace accounts which do have the concept of directories, so all those intermediate directories will be added when uploading the file on portal
—
Reply to this email directly, view it on GitHub<#866 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAEQLMWXIEFFK544B2XAG43VZUIVTANCNFSM562FTTMQ>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
You could manually make the directories using blobfuse or manually upload an empty blob with the metadata key value pair 'hdi_isfolder': 'true' |
So I just create a file without name and add that as content? I will try that asap
Get Outlook for iOS<https://aka.ms/o0ukef>
…________________________________
From: Gauri Prasad ***@***.***>
Sent: Wednesday, August 17, 2022 6:47:15 PM
To: Azure/azure-storage-fuse ***@***.***>
Cc: Xavier Geerinck ***@***.***>; Mention ***@***.***>
Subject: Re: [Azure/azure-storage-fuse] Can't access file in subdir ("No such file or directory" when accessing) (Issue #866)
You could manually make the directories using blobfuse or manually upload an empty blob with the metadata key value pair 'hdi_isfolder': 'true'
—
Reply to this email directly, view it on GitHub<#866 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAEQLMRSNO55B3NF5LVBU2TVZUJRHANCNFSM562FTTMQ>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Sorry, the file name would be whatever virtual directory you are trying to add (for example, a, a/b, a/b/c in your example are the virtual directory names), it's content is empty and the metadata is 'hdi_isfolder': 'true' |
yep that looks correct |
Awesome! Just verified it as well and it seems to work! Thanks a lot :) let's maybe update the documentation somehow to reflect this requirement? Or will there be some code change to automatically create it if it doesn't exist? |
We can definitely update the documentation to reflect this requirement! I will also check with my team and see if this automatic creation is feasible. Thanks again for trying out blobfuse2! We are happy to help as you hit other questions/issues. |
Thanks a lot! Interesting note, when I use it, it works on mount, then after a while the created files disappear and it's not working anymore again |
When you say created files do you mean the ones manually created by you or the ones that show that they are listed on ls? |
The ones created by me, now it seems they only disappeared in the Storage Explorer, when performing In any case, the discovery doesn't seem to work anymore and the original issue returns where I have to manually navigate to each directory to get it working |
It looks like Storage Explorer and Azure Portal have a slightly different UI. Storage Explorer simply shows the folder but Azure Portal will show you the folder and the marker blob, so you could check Azure Portal to double check the marker is indeed gone. Blobfuse doesn't delete any folders unless an explicit call comes from the file system to delete the folder. Would you be able to share any debug logs for this behavior? Or even a set of steps I can try to follow? |
I will debug it a bit more as it is currently training a model now through BlobFuse. It indeed seems that the Azure Portal does show the markers now, so it should work. I will keep you updated on the process! Thanks so much for helping with this so quickly |
Checked more into this and with Hierarchical Namespaces it works better so I am using that one now until a more permanent fix is available :) |
@XavierGeerinck : Just to give you some background on this, block blobs are flat name space file-systems and they do not have any concept of a directory, while blobfuse being a file-system driver has to support directory operations. This leads us to create those special marker files. HNS has native support of directory hence it works there well. Any block blob account where directory markers does not exist, blobfuse will have this limitation as discussed in this thread. |
Thanks a lot for the clarification @vibhansa-msft ! Currently I am trying with HNS support, but am encountering access issues as well :/ it seems that blobfuse sometimes fails to load a file and the entire pipeline crashes (even though the file exist) This is random though, sometimes I have it after 3000 epochs, sometimes after 20000.. so not sure how I can assist in debugging this? |
Can you collect the debug logs for blobfuse, that can help us root-cause the issue. |
We have updated our README file mentioning this directory marker file issue. I will close this issue here as this is by design. If you are facing any other issue, feel free to create a new bug on blobfuse. |
A lot of times these directories are not made by us, so we cannot change them. We still need a way to read them. Currently blobfuse2 stops working randomly. No such issue with blobfuse, so I'm going back to blobfuse for now |
Hi @dashesy, we actually just added a fix for this behavior in the latest code on the main branch. A user needs to set the virtual-directory config parameter to true and virtual directories without markers should work. Could you please try it out? I'd also be happy to share a link to a private build if necessary |
Yes. I will try it out. Please let me know the branch. |
@dashesy It's already merged onto the main branch https://github.com/Azure/azure-storage-fuse/tree/main Please be sure to add the azstorage.virtual-directory config file parameter and set it to true |
@vibhansa-msft I am currently trying to mount ADLS with HNS but am wondering if there is a way to, upon mounting, respect directory-level scope/access using blobfuse? Say my ADLS has a container with 3 directories but my user only has ACL access to 2, is it possible to use blobfuse to mount that container in a way it recognizes my directory-level access and display only what I have access to? |
There are two ways to achieve this.
|
thanks @vibhansa-msft - for 1. how do you go about setting ACL on “others” given that |
chmod is the only option you have to change the ACL for any given file. If your account is not HNS enable or you have mounted using blob endpoint (type: adls is not given in config file) chmod will not be supported. |
Which version of the blobfuse was used?
blobfuse2 latest
Which OS (please include version) are you using?
Linux Ubuntu 20.04
What problem was encountered?
I have a directory structure (e.g. A/B/C/file.txt) but when trying to access the file it does not work. It works after performing an ls on each subdir (e.g. ls A, ls A/B, ls A/B/C and then cat A/B/C/file.txt)
Have you found a mitigation/solution?
It works after performing an ls on each subdir (e.g. ls A, ls A/B, ls A/B/C and then cat A/B/C/file.txt)
By default, blobfuse logs errors to syslog. If this is relevant, is there anything in the syslog that might be helpful?
/
If relevant, please share your mount command.
Mount Command:
blobfuse2 mount /root/azurestorage --read-only --config-file=/docker/azure-blobfuse-config.yaml
Config File:
The text was updated successfully, but these errors were encountered: