-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fs: transfer: don't raise if link failed with ENOSYS #216
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Discovered this while playing with databricks. E.g. `dvc import-url` resulted in: ``` 2023-07-27 01:02:33,446 DEBUG: v3.8.1 (pip), CPython 3.10.6 on Linux-5.15.0-1040-azure-x86_64-with-glibc2.35 2023-07-27 01:02:33,446 DEBUG: command: /local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/bin/dvc import-url https://archive.ics.uci.edu/static/public/186/wine+quality.zip -v 2023-07-27 01:02:36,628 DEBUG: Removing output 'wine+quality.zip' of stage: 'wine+quality.zip.dvc'. 2023-07-27 01:02:36,629 DEBUG: Removing '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/wine+quality.zip' Importing 'https://archive.ics.uci.edu/static/public/186/wine+quality.zip' -> 'wine+quality.zip' 2023-07-27 01:02:37,874 DEBUG: Computed stage: 'wine+quality.zip.dvc' md5: 'bec84f2324d59f520ab3460709b68f87' 2023-07-27 01:02:37,874 DEBUG: 'md5' of stage: 'wine+quality.zip.dvc' changed. 2023-07-27 01:02:44,428 DEBUG: Computed stage: 'wine+quality.zip.dvc' md5: 'ef727310d96dc9022717839b298235be' 2023-07-27 01:02:45,086 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/99fd51b05b33c5f95ca322e09d4465732bac0c61226db94ea7322cb29a00110f' to '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5' 2023-07-27 01:02:45,086 DEBUG: Preparing to collect status from '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5' 2023-07-27 01:02:45,086 DEBUG: Collecting status from '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5' 2023-07-27 01:02:45,386 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/99fd51b05b33c5f95ca322e09d4465732bac0c61226db94ea7322cb29a00110f' 2023-07-27 01:02:45,984 ERROR: failed to transfer '0ddfa7a9379510fe7ff88b9930e3c332' - [Errno 38] Function not implemented: '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/wine+quality.zip' -> '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332' Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 306, in transfer _try_links( File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 250, in _try_links _link(link, from_fs, from_path, to_fs, to_path) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 62, in _link func(from_path, to_path) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 381, in link return self.fs.link(from_info, to_info) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/local.py", line 166, in link return system.hardlink(path1, path2) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/system.py", line 32, in hardlink os.link(src, link_name) OSError: [Errno 38] Function not implemented: '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/wine+quality.zip' -> '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332' 2023-07-27 01:02:45,986 DEBUG: failed to protect '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332' - [Errno 2] No such file or directory: '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332' Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/db/local.py", line 119, in protect os.chmod(path, self.CACHE_MODE) FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332' 2023-07-27 01:02:46,584 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out) 2023-07-27 01:02:46,585 DEBUG: Removing '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.bF8oLrv5ibS4ysuLVnMuhm.tmp' 2023-07-27 01:02:47,446 DEBUG: Removing '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.bF8oLrv5ibS4ysuLVnMuhm.tmp' 2023-07-27 01:02:47,499 DEBUG: Removing '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/.29mHH5XQQ6unJZpZSkL9qD.tmp' 2023-07-27 01:02:49,079 ERROR: failed to import https://archive.ics.uci.edu/static/public/186/wine+quality.zip. You could also try downloading it manually, and adding it with `dvc add`. - unable to remove '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/wine+quality.zip' without a confirmation. Use `-f` to force. Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/output.py", line 759, in _checkout return checkout(*args, **kwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 276, in checkout _checkout( File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 223, in _checkout _checkout_file( File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 88, in _checkout_file _relink( File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 59, in _relink _remove(path, fs, in_cache, force=force, prompt=prompt) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 53, in _remove raise PromptError(path) dvc_data.hashfile.checkout.PromptError: unable to remove '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/wine+quality.zip' without a confirmation. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/commands/imp_url.py", line 15, in run self.repo.imp_url( File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/repo/__init__.py", line 64, in wrapper return f(repo, *args, **kwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 151, in run return method(repo, *args, **kw) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/repo/imp_url.py", line 85, in imp_url stage.run(jobs=jobs, no_download=no_download) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/funcy/decorators.py", line 47, in wrapper return deco(call, *dargs, **dkwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/stage/decorators.py", line 43, in rwlocked return call() File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/funcy/decorators.py", line 68, in __call__ return self._func(*self._args, **self._kwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/stage/__init__.py", line 626, in run self.commit(allow_missing=allow_missing) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/funcy/decorators.py", line 47, in wrapper return deco(call, *dargs, **dkwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/stage/decorators.py", line 43, in rwlocked return call() File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/funcy/decorators.py", line 68, in __call__ return self._func(*self._args, **self._kwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/stage/__init__.py", line 549, in commit out.commit(filter_info=filter_info, **kwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/output.py", line 800, in commit self._checkout( File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/output.py", line 761, in _checkout raise ConfirmRemoveError(exc.path) # noqa: B904 dvc.exceptions.ConfirmRemoveError: unable to remove '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/wine+quality.zip' without a confirmation. Use `-f` to force. 2023-07-27 01:02:49,111 DEBUG: Analytics is enabled. 2023-07-27 01:02:49,314 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpnej7wva2']' 2023-07-27 01:02:49,318 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpnej7wva2']' ```
Codecov ReportPatch coverage has no change and project coverage change:
Additional details and impacted files@@ Coverage Diff @@
## main #216 +/- ##
==========================================
- Coverage 64.79% 63.76% -1.03%
==========================================
Files 25 25
Lines 1943 1943
Branches 308 308
==========================================
- Hits 1259 1239 -20
- Misses 627 648 +21
+ Partials 57 56 -1
☔ View full report in Codecov by Sentry. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Discovered this while playing with databricks. E.g.
dvc import-url
resulted in:And here is a
dvc doctor
for the record: