list: -R fails

[jorgeorpinel]

Bug Report

From https://discuss.dvc.org/t/dvc-list-r-invalidates-repo-path/551

dvc list -R (recursive) fails.

λ cat nest.dvc
outs:
- md5: e795c771edc7a85f3f59e44593c2e9c9.dir
  path: nest
λ tree -A nest
nest
├── 1
│                                   └── 1.dat
└── 2
    │                               └── 2.1
    │                               └── 2.1.dat
    └── 2.dat

3 directories, 3 files

λ dvc list -R . nest
ERROR: failed to list '.' - The path 'nest' does not exist in the target repository '.' neither as a DVC output nor as a Git-tracked file.

Please provide information about your setup

Output of dvc version:

λ dvc version
DVC version: 1.9.1 (exe)
---------------------------------
Platform: Python 3.7.9 on Windows-10-10.0.18362-SP0
Supports: All remotes
Cache types: hardlink
Cache directory: NTFS on C:\
Workspace directory: NTFS on C:\
Repo: dvc, git

Additional Information (if any):

λ dvc list -R . nest/1
ERROR: failed to list '.' - The path 'nest/1' does not exist in the target repository '.' neither as a DVC output nor as a Git-tracked file.
poj12@AP-QDVJ7BLR ~/DVC-repos/test-4734 (master)
λ dvc list -R . nest -v
2020-11-10 12:06:40,637 DEBUG: Creating external repo .@None
2020-11-10 12:06:40,640 DEBUG: erepo: git clone '.' to a temporary dir
2020-11-10 12:06:41,581 DEBUG: cache 'C:\Users\poj12\AppData\Local\Temp\tmpmqppsnaodvc-cache\e7\95c771edc7a85f3f59e44593c2e9c9.dir' expected 'HashInfo(name='md5', value='e795c771edc7a85f3f59e44593c2e9c9.dir', dir_info=None)' actual 'None'
2020-11-10 12:06:41,585 DEBUG: Preparing to download data from '.dvc\cache'
2020-11-10 12:06:41,588 DEBUG: Preparing to collect status from .dvc\cache
2020-11-10 12:06:41,589 DEBUG: Collecting information from local cache...
2020-11-10 12:06:41,593 DEBUG: cache 'C:\Users\poj12\AppData\Local\Temp\tmpmqppsnaodvc-cache\e7\95c771edc7a85f3f59e44593c2e9c9.dir' expected 'HashInfo(name='md5', value='e795c771edc7a85f3f59e44593c2e9c9.dir', dir_info=None)' actual 'None'
2020-11-10 12:06:41,597 DEBUG: Collecting information from remote cache...
2020-11-10 12:06:41,600 DEBUG: Matched '0' indexed hashes
2020-11-10 12:06:41,603 DEBUG: Querying 1 hashes via object_exists
2020-11-10 12:06:41,641 DEBUG: Downloading '.dvc\cache\e7\95c771edc7a85f3f59e44593c2e9c9.dir' to '..\..\AppData\Local\Temp\tmpmqppsnaodvc-cache\e7\95c771edc7a85f3f59e44593c2e9c9.dir'
2020-11-10 12:06:41,663 DEBUG: cache 'C:\Users\poj12\AppData\Local\Temp\tmpmqppsnaodvc-cache\e7\95c771edc7a85f3f59e44593c2e9c9.dir' expected 'HashInfo(name='md5', value='e795c771edc7a85f3f59e44593c2e9c9.dir', dir_info={'1/1.dat': HashInfo(name='md5', value='b026324c6904b2a9cb4b88d6d61c81d1', dir_info=None), '2/2.1/2.1.dat': HashInfo(name='md5', value='26ab0db90d72e28ad0ba1e22ee510510', dir_info=None), '2/2.dat': HashInfo(name='md5', value='26ab0db90d72e28ad0ba1e22ee510510', dir_info=None)})' actual 'HashInfo(name='md5', value='e795c771edc7a85f3f59e44593c2e9c9', dir_info=None)'
2020-11-10 12:06:41,670 DEBUG: Assuming 'C:\Users\poj12\AppData\Local\Temp\tmpmqppsnaodvc-cache\e7\95c771edc7a85f3f59e44593c2e9c9.dir' is unchanged since it is read-only
2020-11-10 12:06:41,674 DEBUG: Assuming 'C:\Users\poj12\AppData\Local\Temp\tmpmqppsnaodvc-cache\e7\95c771edc7a85f3f59e44593c2e9c9.dir' is unchanged since it is read-only
2020-11-10 12:06:41,741 ERROR: failed to list '.' - The path 'nest' does not exist in the target repository '.' neither as a DVC output nor as a Git-tracked file.
------------------------------------------------------------
Traceback (most recent call last):
  File "dvc\command\ls\__init__.py", line 35, in run
  File "dvc\repo\ls.py", line 44, in ls
dvc.exceptions.PathMissingError: The path 'nest' does not exist in the target repository '.' neither as a DVC output nor as a Git-tracked file.
------------------------------------------------------------

[matsui528]

Hi there,

First of all, thanks for dvc! Our lab members always use dvc to manage large files.

Any update on this issue? I am very much looking forward to this issue being resolved. In my use case, I just want to check a part of a big repository before actually pulling it.

For example:

$ tree
.
└── dataset
    ├── cifar.dvc
    └── imagenet.dvc

$ dvc list . -R
.dvcignore
dataset/.gitignore
dataset/cifar.dvc
dataset/cifar/train/1.png
dataset/cifar/train/2.png
dataset/cifar/val/1.png
dataset/cifar/val/2.png
dataset/imagenet.dvc
dataset/imagenet/train/1.png
dataset/imagenet/train/2.png
dataset/imagenet/val/1.png
dataset/imagenet/val/2.png

$ dvc list . dataset/cifar
train
val

The above all commands work as I expected. However, the following command fails:

$ dvc list . dataset/cifar -R
ERROR: failed to list '.' - The path 'dataset/cifar' does not exist in the target repository '/XXX/YYY' neither as a DVC output nor as a Git-tracked file.

I just want the following results:

train/1.png
train/2.png
val/1.png
val/2.png

Best,
Yusuke

[pmrowla]

@matsui528 unfortunately there is no ETA on when this issue will be addressed right now

[matsui528]

I got it, so I'm looking forward to this feature being implemented someday :)

[dberenbaum]

Can't reproduce this one:

$ dvc list -R . cats-dogs
data/train/cats/cat.1.jpg
data/train/cats/cat.10.jpg
data/train/cats/cat.100.jpg
data/train/cats/cat.1000.jpg
data/train/cats/cat.101.jpg
data/train/cats/cat.102.jpg
data/train/cats/cat.103.jpg
...

Closing, but feel free to reopen if I missed something.