status: granular output for directories

[MikkelAntonsen]

Imagine dvc tracking a directory full of images and files containing labels.

If I delete/change/update a label or image, dvc will tell me that something in the directory has changed, but not exactly what. It would be nice is dvc status could be made more granular, returning something like

data.dvc:
    outputs:
        data/foo: new
        data/bar: deleted

instead of

data.dvc:
	changed outs:
		modified:           data

as it is today.

Thanks:)

[efiop]

Discord context https://discordapp.com/channels/485586884165107732/485596304961962003/592632660539604993

[efiop]

Entry points where to start looking at this:

1. Here is where the status is being created https://github.com/iterative/dvc/blob/0.41.3/dvc/output/base.py#L178
2. Need to make Output.changed_checksum() either more granular to report particular files in the directory that has changed or create even a new method that will return a status dict for a directory;

[shcheklein]

we should be careful with this, imagine a directory with 1M new files. Probably we don't want to show all of them. I would say we should show a summary by default or at least on some threshold (on number of changed files per directory)

[jorgeorpinel]

• Once this is implemented (or in parallel), I recommend we deprecate the -R/--recursive option, as dvc status seems to always be recursive (to a certain limit?) already. Just being able to give any dir as target would suffice (full context: status: support outputs as targets [qa] #4191).

[pared]

I agree with @shcheklein. We could probably set some small default threshold of files to show (like 5, to not pollute the terminal) and add option to show all changes.
In case of default dir status I wouldn't display it as file -> status mapping, but rather
status -> list of files with that status just to save some space (in case of explicit show-all-changes it makes more sense to me to show file -> status mapping, as one probably does not want to scroll through the terminal to see the status of a particular file.

[jorgeorpinel]

We could probably set some small default threshold of files to show (like 5

With pagination?

I wouldn't display it as file -> status mapping, but rather status -> list of files

Agree. I think we do this in other commands already like metrics/params show/diff? And other output

[pared]

With pagination?

I would refrain from that in the default case. By default, I think it's better to "show the status of dir, but if it's small enough, we can go for particular files" rather than make user interact with it.

Agree. I think we do this in other commands already like metrics/params show/diff? And other output

Yes, we do, I was just referring to @MikkelAntonsen example.

[lefos99]

What is the progress of this feature request? 😃

[efiop]

@lefos99 Not implemented yet :( To implement it, we'll need to change

dvc/dvc/output/base.py

Line 204 in 79e8e4e

if self.changed_checksum():

to compare self.dir_cache with self.get_checksum().dir_info(both are just lists of simple dicts) and provide granular status instead of a current generic one.

[lefos99]

@efiop Thanks for the update! 😃

[efiop]

Update: we've been revisiting our data management this year (see dvc/objects) and the approach suggested above (with dir_info) is no longer relevant. We will be migrating status to the new objects in the upcoming weeks and will likely implement this along the way.

[dberenbaum]

Closing as completed in #7943. Thanks @skshetry!

[daavoo]

For reference: https://dvc.org/doc/command-reference/data/status#example-granular-output