-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dvc move fails when cache dir is read-only #4232
Comments
Hi @florianspecker ! Could you please remove duplicated |
@florianspecker What are the permissions on |
Hi @efiop thanks a lot for getting back to me so quickly! I went through all the details again, and embarrassingly enough, I've made a mess of my different folders (the issue showed first in a folder with sensitive customer data, so I reproduced it with a less critical dataset). Sorry for that - I updated the output and config above. Regarding permissions: /tmp/dvc-data is a NFS read-only mount. My user has enough permissions to list and read everything, but writing is not possible.
It looks like the problem is caused by attempting to write |
@florianspecker Nice catch! Indeed, looks like the cause. We can definitely handle it better. As a workaround, could you create that file by-hand(contents don't matter, could even create an empty one) and try again? |
@efiop thanks! Sure, no problem - now
|
@florianspecker Thanks! So no workaround for now other than making it not read-only 🙁 Need to take a closer look and solve properly. Is it blocking you right now or are you able to use a non-read-only fs for now? |
Our users can't switch the file system to read/write. But it's not super urgent, as there are not many move operations, and we can always move manually (i.e. move the image, move the .dvc file, remove entries in the source .gitignore, add entries to the destination .gitignore). |
Got the same error when doing a dvc pull: I can try to bissect this:
|
@clementperon Sorry for the delay. Is '/home/clement/XXX/.dvc/remote-cache/` the dvc remote from which you are trying to pull? |
@dberenbaum yes it's a mount point either sshfs or nfs mounted in Read-only. I shared data throurgh multiple clients, each client mount the data folder in sshfs or nfs and I don't want them to write in this folder. Unfortunately dvc try to write a tmp file in this folder... Is it possible to change the location where this tmp file is located ? mount point: SSHFS |
Unfortunately I'm not aware of any workarounds at the moment. |
For how we use dvc, staying with version 2.1.0 does the trick. It's slow, but at least it works. |
@florianspecker Is there some regression you have noted after 2.1.0? Your original issue was with 1.1.10. Was it fixed at some point? |
@dberenbaum yes with later versions it's not just |
/Volumes/dvc is a read-only NFS mount. The issue looks very similar to #3510
Background on our setup:
DVC pushes images to S3. Our dataset engineers have the .dvc/config as pasted above. The same config is used for our ML training runs that are being executed on AWS.
Our ML developers need the same datasets, and so do the ML trainings that are executed onprem (we have our own GPUs; AWS is only used for peaks). Both these use cases have .dvc/config.local as pasted above (in addition to the same .dvc/config as everybody else). It points to a NFS share, where we sync the content of our S3 bucket. It is read-only to make sure it stays consistent.
Environment:
DVC version: 1.1.10
Python version: 3.8.4
Platform: macOS-10.15.5-x86_64-i386-64bit
Binary: False
Package: brew
Supported remotes: azure, gdrive, gs, http, https, s3, ssh, oss
Repo: dvc, git
The text was updated successfully, but these errors were encountered: