status: support outputs as targets [qa]

[jorgeorpinel]

Bug Report

UPDATE: Jump to #4191 (comment)

---

Looks like dvc status is recursive by default now (maybe always was) from some testing I just did:

λ dvc status
foo.dvc:
        changed outs:
                modified:           foo
data\raw.dvc:
        changed outs:
                modified:           data\raw

λ dvc status -R data/
data\raw.dvc:
        changed outs:
                modified:           data\raw

1. So it's really just useful to limit the status to specific directories (not really about recursion).
2. Why not just support dirs AND FILES as targets of the command, like all other commands that take targets (I think)?

Relater to iterative/dvc.org#1384 (comment)

3. In fact this is already supported by status, but only in remote mode (-r or -c options), which is kind of confusing (and complicates the docs).

Please provide information about your setup

DVC version: 1.1.2
Python version: 3.7.5
Platform: Windows-10-10.0.18362-SP0
Binary: True
Package: exe
Supported remotes: azure, gdrive, gs, hdfs, http, https, s3, ssh, oss
Cache: reflink - not supported, hardlink - supported, symlink - not supported
Filesystem type (cache directory): ('NTFS', 'C:\\')
Repo: dvc, git
Filesystem type (workspace): ('NTFS', 'C:\\')

[efiop]

@jorgeorpinel Non -c status just doesn't support granularity right now.

[efiop]

Closing in favor of #2180

[efiop]

To clarify:

Why not just support dirs AND FILES as targets of the command, like all other commands that take targets (I think)?

It does support files as targets, but not files(or subdirs) in tracked directories (this is what -c/push/pull/etc support).

[jorgeorpinel]

It does support files as targets

@efiop so is this a bug? 👇

λ ls foo*
foo  foo.dvc
λ dvc status foo
ERROR: failed to obtain data status - 'dvc.yaml' does not exist.

[efiop]

@jorgeorpinel Yes, looks like a bug. Reopening. Thanks!

[jorgeorpinel]

Another related question: dvc repro also has targets but only accepts stage and .dvc file names, right? (Otherwise, -R would also seem obsolete there.) If so, should it accept file/dir names (and support granularity)? Prob not.

[efiop]

@jorgeorpinel -R is not obsolete, it searches recursively for dvc files. But when you specify target explicitly, it finds 1 dvc file that it belongs too, so it is not the same.

[jorgeorpinel]

OK thanks. So dvc repro also supports tracked files and thus granularity already? Will double check and include in #1384 then.

What about run, (un)freeze, remove, unprotect, update, and metrics/plots show? Will ask in the appropriate issue...

[efiop]

@jorgeorpinel tracked files and granularity is not the same thing. When we were talking about push/pull we were talking about being able to pull specific file (or subdir) in a tracked dataset (dvc add data_dir). In case of repro we support addressing stages by outputs, which is a different thing.

[jorgeorpinel]

Got it. So I think we need to clarify when file/dir targets support granularity and when they don't, after all. I'll add a note in all the sync-related ones (in #1384) both in the description and in Specific target examples.

[karajan1001]

It does support files as targets

@efiop so is this a bug? 👇

λ ls foo*
foo  foo.dvc
λ dvc status foo
ERROR: failed to obtain data status - 'dvc.yaml' does not exist.

@jorgeorpinel , @efiop
Supporting outputs as targets is a feature, not a bug I think?

Targets in dvc status are stages just the same as what in dvc repo. But the error message ERROR: failed to obtain data status - 'dvc.yaml' does not exist. is a bug. In a previous version (0.94.1), it used to be

So the solution is either to support outputs as targets or to improve the message.

[efiop]

@karajan1001 Adjusted the labels, thanks! 🙂

Since 1.0 we've changed the defaults, hence why it looks for dvc.yaml first. targets are shared by status and status -c, hence the confusion.

[karajan1001]

@efiop
In a local status mode, we use Repo.collect to collect stages

dvc/dvc/repo/status.py

Lines 28 to 33 in 32b5b33

	def _local_status(self, targets=None, with_deps=False, recursive=False):
	if targets:
	stages = cat(
	self.collect(t, with_deps=with_deps, recursive=recursive)
	for t in targets
	)

While in a cloud status mode we are using Repo.collect_granular

dvc/dvc/repo/__init__.py

Lines 378 to 382 in 32b5b33

	pairs = cat(
	self.collect_granular(
	target, recursive=recursive, with_deps=with_deps
	)
	for target in targets

dvc/dvc/repo/__init__.py

Lines 340 to 352 in 32b5b33

	def used_cache(
	self,
	targets=None,
	all_branches=False,
	with_deps=False,
	all_tags=False,
	all_commits=False,
	remote=None,
	force=False,
	jobs=None,
	recursive=False,
	used_run_cache=None,
	):

dvc/dvc/repo/status.py

Lines 79 to 89 in 32b5b33

	used = self.used_cache(
	targets,
	all_branches=all_branches,
	all_tags=all_tags,
	all_commits=all_commits,
	with_deps=with_deps,
	force=True,
	remote=remote,
	jobs=jobs,
	recursive=recursive,
	)

dvc/dvc/repo/status.py

Lines 40 to 50 in 32b5b33

	def _cloud_status(
	self,
	targets=None,
	jobs=None,
	remote=None,
	all_branches=False,
	with_deps=False,
	all_tags=False,
	recursive=False,
	all_commits=False,
	):

And if the one stage and one output have the same name, the stage would win. This will prevent the users from selecting those outputs which have the same name with stages. (Before version 1.0, stages name will always have a .dvc suffix which prevents this problem)

[karajan1001]

dvc/dvc/repo/__init__.py

Lines 302 to 304 in 32b5b33

	# parsing is ambiguous when it does not have a colon
	# or if it's not a dvcfile, as it can be a stage name
	# in `dvc.yaml` or, an output in a stage.

Wow, had been considered before. stage is preferred to output

[skshetry]

@karajan1001, there's a workaround: dvc status ./<filename> for now. 🙂

But, clearly, it's not implemented for status, only for -c.

[skshetry]

And, we don't recommend to have a stage name same as outputs.