-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unexpected error - Adding files #2831
Comments
Hi @luchoPipe87 ! That looks pretty weird. Btw, why did you remove fs info from |
Hi @efiop! |
@luchoPipe87 You are not working on a NAS partition or anything like that? Just regular hard drive(ssd) on your machine? Btw, did you run |
I get the same error when doing: dvc init
dvc destroy -f |
@casperdcl I am able to reproduce from that too! Thank you! Looks like it is our daemon getting confused by a deleted lock. Not quite sure what is up with |
I'm doing |
@luchoPipe87 Is there any other dvc process running in the background? |
@luchoPipe87 Got it. And does running simple |
If i do
And i received this one after a
|
@luchoPipe87 And could you try manually deleting |
every time that i deleted
|
@luchoPipe87 Interesting. And what flufl.lock version do you have installed? Could you show |
@luchoPipe87 Right, sorry, I thought that you are using gitbash 🙂 Not sure what is the alternative command for cmd/powershell. Ok, so flufl lock version looks alright. Ok, let's try to narrow this down. Could you please try running this script in your repo root:
|
No worries! :)
|
|
@luchoPipe87 Thanks! Ok, looks like it might be flufl.lock issue. Could you try this too, please:
|
@luchoPipe87 Ok, this might be the issue https://gitlab.com/warsaw/flufl.lock/blob/master/flufl/lock/_lockfile.py#L490 , because stat on windows is pretty useless IIRC. Could you try running:
|
As one could see https://github.com/iterative/dvc/blob/master/dvc/system.py#L280 we actually don't use os.stat to count the number of hardlinks. flufl.lock should do the same, but the project is pretty much dead. 'Guess we should consider forking it into something like flufl.lock2 and fix up these kind of things. But we should definitely start from simply patching it up in our Lock class. |
@luchoPipe87 I'll prepare a patch in a minute and will ask you to try it out, if you don't mind 🙂 |
@luchoPipe87 Ok, please install my version of dvc with
and try running those commands again. |
Here the output of the last command :)
|
@luchoPipe87 You mean that test script with the patched dvc? Could you run |
I installed your dvc version and its getting better (Is another error :P )
|
@luchoPipe87 I've updated my patch, please reinstall with
and try |
In my case:
|
@MrOutis It was windows 10 :) |
@efiop, but there are different versions of Windows 10, right? I haven't read the release log, but might be a driver thingy? https://docs.microsoft.com/en-us/windows/release-information/ |
@MrOutis It was provided in the |
@luchoPipe87 Hm, btw, do you have a RAID or something setup on your machine? Maybe some special encryption stuff? I just wonder if the link we are creating get's altered by something right away, maybe some weird encryption stuff or backups... Considering that it is work machine, i suppose it has both raid/backups and some special encryption. Or maybe antivirus O_o? |
@luchoPipe87 ping 🙂 |
hi @efiop ! |
@luchoPipe87 No worries 🙂 I still don't have a certain explanation for what is going on your machine, but the workaround that I've provided you with should still work and it will soon become the default behavior (see #2918). |
As it turned out (see issue numbers down below), we can't really take hardlinks for granted, so `flufl.lock` is not a panacea for all filesystems. Considering that the vast majority of filesystems that our users use support `zc.lockfile`(flock-based) and it has benefits like more reliable mechanism, auto-delete when process exits, more sturdy implementation, etc, it makes more sense to bring it back and use by default again. For filesystems that don't support `flock()`, users will be able to manually enable `flufl.lock` use through the config option. It would be ideal if we could auto-detect that flock is not supported, but in the real world, it turned out to be non-trivial, as it might hang forever in a kernel context, which makes the implementation way too complex for our purposes. So what we're doing instead is showing a message before locking with `zc.lockfile` that, under normal circumstances will disappear once the lock is taken or failed, otherwise it will point users to the related documentation where they can learn about how to opt-in for `flufl.lock`. Fixes iterative#2831 Fixes iterative#2897 Related iterative#2860
Hi @luchoPipe87 ! Another user is having a similar strange hardlink behavior and we've created a patch that might solve it. If you have time, could you please run commands described in #3080 (comment) and show us the output of the last one, please? Thanks! |
Hi!
|
Thanks @luchoPipe87 ! 🙏 Could you please create |
hi, $dvc version Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help! |
@luchoPipe87 Thanks! Ok, so it looks like your case is indeed a bit different. Hardlinks are created, but there is no way for us to count them to verify it. Very interesting. Still not sure what we could do there to fix it, but at least it works for you now with flock() based locks. Btw, did you ask your sys admin about it by any chance? Also, how is your dvc experience so far? 🙂 |
Everytime that im trying to add some individuals files or complete directories the same unexpected error appears:
I have a remote configuration by SSH:
And here the version of dvc that im using:
Thanks for your help!
The text was updated successfully, but these errors were encountered: