Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DVC tracked file rename fails when file is not pulled yet #808

Closed
shcheklein opened this issue Sep 14, 2021 · 4 comments · Fixed by #5221
Closed

DVC tracked file rename fails when file is not pulled yet #808

shcheklein opened this issue Sep 14, 2021 · 4 comments · Fixed by #5221
Assignees
Labels
A: integration Area: DVC integration layer A: trees Area: SCM and DVC-tracked trees bug Something isn't working priority-p2 Future feature, less priority for now

Comments

@shcheklein
Copy link
Member

shcheklein commented Sep 14, 2021

After trying to rename the images in the get-started-experiment:

Screen Shot 2021-09-13 at 10 41 58 PM

File and directory are completely missing in the DVC tracked view.

Logs ``` ERROR: failed to get diff - Unable to acquire lock. Most likely another DVC process is running or was terminated abruptly. Check the page for other possible reasons and to learn how to resolve this. [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:30:10.240Z, pid: 8062] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc status --show-json - COMPLETED (7189ms) [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:30:34.872Z, pid: 8071] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc list . --dvc-only -R --show-json - COMPLETED (31813ms) [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:34:09.044Z, pid: 9486] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc list . data --dvc-only --show-json - INITIALIZED [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:34:09.572Z, pid: 9486] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc list . data --dvc-only --show-json - COMPLETED (530ms) [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:35:26.889Z, pid: 10054] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc pull data/images.tar.gz - INITIALIZED [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:35:29.514Z, pid: 10054] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc pull data/images.tar.gz - FAILED with code 255 (2629ms) ERROR: unexpected error - [Errno 22] The provided token is malformed or otherwise invalid.: An error occurred (InvalidToken) when calling the ListObjectsV2 operation: The provided token is malformed or otherwise invalid. [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:39:06.877Z, pid: 11773] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc diff --show-json - INITIALIZED [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:39:06.879Z, pid: 11774] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc status --show-json - INITIALIZED [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:39:06.887Z, pid: 11783] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc list . --dvc-only -R --show-json - INITIALIZED [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:39:09.817Z, pid: 11773] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc diff --show-json - FAILED with code 1 (2944ms) ERROR: failed to get diff - Unable to acquire lock. Most likely another DVC process is running or was terminated abruptly. Check the page for other possible reasons and to learn how to resolve this. [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:39:14.659Z, pid: 11774] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc status --show-json - COMPLETED (7782ms) [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:40:05.360Z, pid: 11783] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc list . --dvc-only -R --show-json - COMPLETED (58476ms) [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:40:18.505Z, pid: 12299] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc move data/images.tar.gz data/images1.tar.gz - INITIALIZED [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:40:19.518Z, pid: 12326] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc list . data --dvc-only --show-json - INITIALIZED [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:40:20.648Z, pid: 12326] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc list . data --dvc-only --show-json - FAILED with code 1 (1132ms) ERROR: failed to list '.' - The path 'data' does not exist in the target repository '/Users/ivan/Projects/get-started-experiments' as an DVC output. [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:40:20.681Z, pid: 12331] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc list . --dvc-only --show-json - INITIALIZED [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:40:21.427Z, pid: 12299] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc move data/images.tar.gz data/images1.tar.gz - FAILED with code 255 (2925ms) ERROR: unexpected error - [Errno 2] No such file or directory: '/Users/ivan/Projects/get-started-experiments/data/images.tar.gz' [version: 0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0, 2021-09-14T05:40:21.938Z, pid: 12331] > /Users/ivan/Projects/get-started-experiments/.env/bin/python -m dvc list . --dvc-only --show-json - COMPLETED (1260ms) ```
Environment

0.1.0-e1bf948c5cc02edf6260c675401d500502abe9b0

Version: 1.60.0
Commit: e7d7e9a9348e6a8cc8c03f877d39cb72e5dfb1ff
Date: 2021-09-01T10:54:53.442Z
Electron: 13.1.8
Chrome: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Darwin x64 19.6.0
DVC version: 2.7.3+788b4d 
---------------------------------
Platform: Python 3.8.9 on macOS-10.15.6-x86_64-i386-64bit
Supports:
        azure (adlfs = 2021.9.1, knack = 0.8.2, azure-identity = 1.6.0),
        gdrive (pydrive2 = 1.9.3),
        gs (gcsfs = 2021.8.1),
        hdfs (pyarrow = 4.0.1),
        webhdfs (hdfs = 2.6.0),
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
        s3 (s3fs = 2021.8.1, boto3 = 1.17.106),
        ssh (sshfs = 2021.8.1),
        oss (ossfs = 2021.8.0),
        webdav (webdav4 = 0.9.1),
        webdavs (webdav4 = 0.9.1)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s1
Repo: dvc, git
@shcheklein
Copy link
Member Author

Overall, rename command is very complicated in DVC and move doesn't implement that semantics properly. It's another item that we'll need to discuss and prioritize on the DVC end (cc @efiop @dberenbaum ).

We'll need to being able to rename model files (outputs of the pipelines). Or don't allow that action in the first place as a simple workaround.

@shcheklein shcheklein added bug Something isn't working A: trees Area: SCM and DVC-tracked trees labels Sep 14, 2021
@mattseddon
Copy link
Member

Relates to #569

@shcheklein
Copy link
Member Author

First workaround to try: disable this in situations that DVC can't handle (e.g. when files are not physically present yet).

@shcheklein shcheklein added the priority-p1 Regular product backlog label Aug 23, 2022
@mattseddon mattseddon added blocked Issue or pull request blocked due to other dependencies or issues and removed blocked Issue or pull request blocked due to other dependencies or issues labels Oct 18, 2022
@mattseddon mattseddon added priority-p2 Future feature, less priority for now and removed priority-p1 Regular product backlog labels Dec 1, 2022
@mattseddon mattseddon added priority-p2 Future feature, less priority for now and removed priority-p2 Future feature, less priority for now labels Mar 21, 2023
@mattseddon
Copy link
Member

Let's delete the rename command.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: integration Area: DVC integration layer A: trees Area: SCM and DVC-tracked trees bug Something isn't working priority-p2 Future feature, less priority for now
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants