-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dvc: make flufl.lock opt-in and use zc.lockfile #2918
Conversation
@@ -79,6 +79,7 @@ def run(self): | |||
"shortuuid>=0.5.0", | |||
"tqdm>=4.40.0,<5", | |||
"packaging>=19.0", | |||
"zc.lockfile>=1.2.1", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note to self: no need to update this in our conda package, as I've left zc.lockfile there in preparation for this PR.
dvc/updater.py
Outdated
@@ -24,11 +23,13 @@ class Updater(object): # pragma: no cover | |||
TIMEOUT = 24 * 60 * 60 # every day | |||
TIMEOUT_GET = 10 | |||
|
|||
def __init__(self, dvc_dir): | |||
def __init__(self, dvc_dir, **kwargs): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would name it explicitly, this is shadowing the intent and tying up the interfaces.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good to me, @efiop ;
@efiop @jorgeorpinel I've just put a few minor UI comments, please take a look and improve/fix if needed. |
@efiop btw, just curious, how does it affect |
@shcheklein |
As it turned out (see issue numbers down below), we can't really take hardlinks for granted, so `flufl.lock` is not a panacea for all filesystems. Considering that the vast majority of filesystems that our users use support `zc.lockfile`(flock-based) and it has benefits like more reliable mechanism, auto-delete when process exits, more sturdy implementation, etc, it makes more sense to bring it back and use by default again. For filesystems that don't support `flock()`, users will be able to manually enable `flufl.lock` use through the config option. It would be ideal if we could auto-detect that flock is not supported, but in the real world, it turned out to be non-trivial, as it might hang forever in a kernel context, which makes the implementation way too complex for our purposes. So what we're doing instead is showing a message before locking with `zc.lockfile` that, under normal circumstances will disappear once the lock is taken or failed, otherwise it will point users to the related documentation where they can learn about how to opt-in for `flufl.lock`. Fixes iterative#2831 Fixes iterative#2897 Related iterative#2860
@efiop does config have |
Thanks @pared for catching this one #2918 (comment) . Will add a test on top, need to re-release ASAP
As it turned out (see issue numbers down below), we can't really take
hardlinks for granted, so
flufl.lock
is not a panacea for allfilesystems. Considering that the vast majority of filesystems that our
users use support
zc.lockfile
(flock-based) and it has benefits likemore reliable mechanism, auto-delete when process exits, more sturdy
implementation, etc, it makes more sense to bring it back and use by
default again. For filesystems that don't support
flock()
, users willbe able to manually enable
flufl.lock
use through the config option.It would be ideal if we could auto-detect that flock is not supported,
but in the real world, it turned out to be non-trivial, as it might hang
forever in a kernel context, which makes the implementation way too
complex for our purposes. So what we're doing instead is showing a
message before locking with
zc.lockfile
that, under normalcircumstances will disappear once the lock is taken or failed, otherwise
it will point users to the related documentation where they can learn
about how to opt-in for
flufl.lock
.Fixes #2831
Fixes #2897
Fixes #2944
Related #2860
❗ Have you followed the guidelines in the Contributing to DVC list?
📖 Check this box if this PR does not require documentation updates, or if it does and you have created a separate PR in dvc.org with such updates (or at least opened an issue about it in that repo). Please link below to your PR (or issue) in the dvc.org repo.
iterative/dvc.org#858
Thank you for the contribution - we'll try to review it as soon as possible. 🙏