dvc: support granularity for fetch/pull/push/status/checkout

[efiop]

Using output path or a subdir/subfile path within an output now works
with dvc push/pull/fetch/status -c commands. Other commands don't
support the same logic for now, as there are some questions about what
should commands like dvc remove do when given a specific output path.

Example:
dvc add data
dvc pull data/subdir # will only pull files within data/subdir

Related to #2458

• ❗ Have you followed the guidelines in the Contributing to DVC list?

• 📖 Check this box if this PR does not require documentation updates, or if it does and you have created a separate PR in dvc.org with such updates (or at least opened an issue about it in that repo). Please link below to your PR (or issue) in the dvc.org repo.

iterative/dvc.org#886

• ❌ Have you checked DeepSource, CodeClimate, and other sanity checks below? We consider their findings recommendatory and don't expect everything to be addressed. Please review them carefully and fix those that actually improve code or fix bugs.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

[Suor]

How about a new plan?

• collect outs not stages, i.e. drop ._collect_granular(), make .collect() return a list of outs
• move needed logic from stage to out (out knows its stage, so should be straitforward)
• use out.checkout(), out.get_files_number(), ... where appropriate instead of same stage methods
• create FilteredOutput(out, path_info) wrapper to support granularity within out dirs, expose all needed methods. Make these in Repo.collect() when needed.
• drop no longer used Stage methods, things are done on output level now

I see some rwlock complications, but again rwlock is also output based ;)

In the end everything should look leaner than now not more complicated. I hope)

P.S. On rwlock complications. We might move it to out level, but that will make us constantly updating locks. The alternative is to move it higher level and lock everything at repo level after we resolved targets to outs/path_infos we may lock them all at once before doing things. Not sure which one is better.

[efiop]

@Suor Have thought about that, but it will require quite a heavy refactoring. I would rather not get into it right now. Plus, outputs are not atomic, as we have directories and we need to support granularity for files inside of them, so FilteredOutput won't work as-is.

[Suor]

@efiop don't get why FilteredOutput() won't work, how is this connected with atomicity?

[Suor]

Now a lot of methods on stage simply pass things into outs and back, this is the sign. And adding path_info filter to Stage methods doesn't look pretty either.

[efiop]

@efiop don't get why FilteredOutput() won't work, how is this connected with atomicity?

I see what you mean. Yeah, could do that.

Now a lot of methods on stage simply pass things into outs and back, this is the sign. And adding path_info filter to Stage methods doesn't look pretty either.

It has been a sign for a long time, this PR is not meant to deal with that. Need to create a separate ticket and deal with it as refactoring.

[efiop]

@Suor I'm still dealing with some checkout parts, will see if the refactoring you suggest will help with that. E.g. get_files_number thingy might be replaced by a dynamic pbar update, which is more reasonable than the giant refactoring that you suggest 🙂

[Suor]

OK, even without the big refactoring you can make Repo.collect() return a list of outs, some of which will be filtered. Then do the out.stage.do_thing(..., path_info=out.path_info) when you actually want to do stuff, which doesn't have a method on out. This way you won't need pairs.

Hmm. Actually we have two types of operations stage and out ones. It's repro and everything else correspondingly. So we might want repo.collect_stages() and repo.collect_outs(). Or should we call them .resolve_stages(targets) and .resolve_outs(targets)?

Also, I don't perceive this as a giant refactoring, just a normal non-small one ))

[efiop]

@Suor I appreciate the ideas and I have considered this while working on this PR, but I would prefer to begin with this change and then proceed with the refactoring, as putting it into this PR will only blow it out of proportion.

[Suor]

I feel how our debt is growing (

[Suor]

I wonder how much slower is this.

[efiop]

I feel how our debt is growing (

@Suor Yeah 🙁 I think we could do better than FilteredOutput. It seems like our output dir and file concepts are a bit off and we actually need OutputDir and Output(File), that OutputDir will consist of. That way all our manipulations with get_files_number and progress bars would be much more straightforward, as checkout would be always dealing with Output(File)s, which are trivial to wrap in tqdm pbar. Also we would simply filter right away, instead of creating a special FilteredOutput class. Though unpacked tricks would probably cause some troubles there. That being said, working on a pre-collected list of output files, would also simplify parallelization and stuff like that, so we might get rid of unpacked_dir for good. Need to take a closer look.

[Suor]

So there is a last question )