Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fs: transfer: don't raise if link failed with ENOSYS
Discovered this while playing with databricks. E.g. `dvc import-url` resulted in: ``` 2023-07-27 01:02:33,446 DEBUG: v3.8.1 (pip), CPython 3.10.6 on Linux-5.15.0-1040-azure-x86_64-with-glibc2.35 2023-07-27 01:02:33,446 DEBUG: command: /local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/bin/dvc import-url https://archive.ics.uci.edu/static/public/186/wine+quality.zip -v 2023-07-27 01:02:36,628 DEBUG: Removing output 'wine+quality.zip' of stage: 'wine+quality.zip.dvc'. 2023-07-27 01:02:36,629 DEBUG: Removing '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/wine+quality.zip' Importing 'https://archive.ics.uci.edu/static/public/186/wine+quality.zip' -> 'wine+quality.zip' 2023-07-27 01:02:37,874 DEBUG: Computed stage: 'wine+quality.zip.dvc' md5: 'bec84f2324d59f520ab3460709b68f87' 2023-07-27 01:02:37,874 DEBUG: 'md5' of stage: 'wine+quality.zip.dvc' changed. 2023-07-27 01:02:44,428 DEBUG: Computed stage: 'wine+quality.zip.dvc' md5: 'ef727310d96dc9022717839b298235be' 2023-07-27 01:02:45,086 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/99fd51b05b33c5f95ca322e09d4465732bac0c61226db94ea7322cb29a00110f' to '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5' 2023-07-27 01:02:45,086 DEBUG: Preparing to collect status from '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5' 2023-07-27 01:02:45,086 DEBUG: Collecting status from '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5' 2023-07-27 01:02:45,386 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/99fd51b05b33c5f95ca322e09d4465732bac0c61226db94ea7322cb29a00110f' 2023-07-27 01:02:45,984 ERROR: failed to transfer '0ddfa7a9379510fe7ff88b9930e3c332' - [Errno 38] Function not implemented: '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/wine+quality.zip' -> '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332' Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 306, in transfer _try_links( File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 250, in _try_links _link(link, from_fs, from_path, to_fs, to_path) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 62, in _link func(from_path, to_path) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 381, in link return self.fs.link(from_info, to_info) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/local.py", line 166, in link return system.hardlink(path1, path2) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/system.py", line 32, in hardlink os.link(src, link_name) OSError: [Errno 38] Function not implemented: '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/wine+quality.zip' -> '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332' 2023-07-27 01:02:45,986 DEBUG: failed to protect '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332' - [Errno 2] No such file or directory: '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332' Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/db/local.py", line 119, in protect os.chmod(path, self.CACHE_MODE) FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332' 2023-07-27 01:02:46,584 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out) 2023-07-27 01:02:46,585 DEBUG: Removing '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/.bF8oLrv5ibS4ysuLVnMuhm.tmp' 2023-07-27 01:02:47,446 DEBUG: Removing '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/.bF8oLrv5ibS4ysuLVnMuhm.tmp' 2023-07-27 01:02:47,499 DEBUG: Removing '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/.29mHH5XQQ6unJZpZSkL9qD.tmp' 2023-07-27 01:02:49,079 ERROR: failed to import https://archive.ics.uci.edu/static/public/186/wine+quality.zip. You could also try downloading it manually, and adding it with `dvc add`. - unable to remove '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/wine+quality.zip' without a confirmation. Use `-f` to force. Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/output.py", line 759, in _checkout return checkout(*args, **kwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 276, in checkout _checkout( File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 223, in _checkout _checkout_file( File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 88, in _checkout_file _relink( File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 59, in _relink _remove(path, fs, in_cache, force=force, prompt=prompt) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 53, in _remove raise PromptError(path) dvc_data.hashfile.checkout.PromptError: unable to remove '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/wine+quality.zip' without a confirmation. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/commands/imp_url.py", line 15, in run self.repo.imp_url( File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/repo/__init__.py", line 64, in wrapper return f(repo, *args, **kwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 151, in run return method(repo, *args, **kw) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/repo/imp_url.py", line 85, in imp_url stage.run(jobs=jobs, no_download=no_download) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/funcy/decorators.py", line 47, in wrapper return deco(call, *dargs, **dkwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/stage/decorators.py", line 43, in rwlocked return call() File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/funcy/decorators.py", line 68, in __call__ return self._func(*self._args, **self._kwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/stage/__init__.py", line 626, in run self.commit(allow_missing=allow_missing) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/funcy/decorators.py", line 47, in wrapper return deco(call, *dargs, **dkwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/stage/decorators.py", line 43, in rwlocked return call() File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/funcy/decorators.py", line 68, in __call__ return self._func(*self._args, **self._kwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/stage/__init__.py", line 549, in commit out.commit(filter_info=filter_info, **kwargs) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/output.py", line 800, in commit self._checkout( File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/output.py", line 761, in _checkout raise ConfirmRemoveError(exc.path) # noqa: B904 dvc.exceptions.ConfirmRemoveError: unable to remove '/Workspace/Repos/[email protected]/databricks-mlflow-end-to-end-example/wine+quality.zip' without a confirmation. Use `-f` to force. 2023-07-27 01:02:49,111 DEBUG: Analytics is enabled. 2023-07-27 01:02:49,314 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpnej7wva2']' 2023-07-27 01:02:49,318 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpnej7wva2']' ```
- Loading branch information