Add stages command

[skshetry]

Resolves #3743.

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.
  Not looking into adding a docs yet, keeping this command as experimental for some time.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

[pared]

@skshetry do we intend to have tests for stages?

[skshetry]

@pared, the autocompletion-related feature stages -sq is a very thin wrapper on dvc.repo.stage. The long-form list output is mostly presentational with very little logic and is experimental for now. I'm sure this will change. Till now, I was just toying with rich and its feature, so I haven't thought much about the information that we should display there.

The feedback on that and the design would be very much appreciated. Thanks.

[pared]

@skshetry Thanks,
So far it seems very clear and understandable. I'll play around with it and see if I can come up with something useful.

[pared]

Support for markdown in description looks great!

[pmrowla]

LGTM.

Only comment would be that having the entire tabled centered instead of left-aligned looks off to me in a wider terminal window, but that's a UI personal preference thing.

[shcheklein]

Looks great overall! I have a few concerns though to be honest:

1. A separate command - it feels if overlaps with dvc dag to some extent. Even if not, do we actually need the whole new command for this? Should we merge dag into this one dvc stages --dag?
2. Interface looks refreshing, but a bit too massive to my mind. Feels like -s can be default? And even if we add more details - let's add them as ls does (compact table, parsable with regular tools like awk - it's quite important to my mind)
3. It's better to always provide --show-json, or something similar for machine readable interface. So that UI does not become an API (as much as possible). In case awk parsable table is possible, it might we okay to do both?

[skshetry]

1. A separate command - it feels if overlaps with `dvc dag` to some extent. Even if not, do we actually need the whole new command for this? Should we merge `dag` into this one `dvc stages --dag`?

dag is more of a visual command whereas my intention for this command is to just list stages in the dvc.yaml in a quick and easy way (the dependent nodes are only shown if they are within the same file, and graph collections are skipped to make it faster). Anything other than that is optional, the command is only trying to be helpful.

There was a discussion on showing status here or move this to the dvc status, as status is limited, and this stages format provides an opportunity to provide better information (eg: with granularity and dag). But I am not sure about this command in the longer term or whether we should move them to status/dag or move them here.

2. Interface looks refreshing, but a bit too massive to my mind. Feels like `-s` can be default? 

Could that be the other way around? Encourage people to use --short/-s for parsing/grepping?

And even if we add more details - let's add them as ls does (compact table, parsable with regular tools like awk - it's quite important to my mind)

ls might not work as the columns might vary based on description/outputs/dependent nodes/dependencies, etc.
Unless we choose to not show them at all and just show the numbers of them, it will be difficult to parse anyway.

3. It's better to always provide `--show-json`, or something similar for machine readable interface. So that UI does not become an API (as much as possible). In case `awk` parsable table is possible, it might we okay to do both?

-s should be machine-readable. The current format that's required for autocompletion could be changed to a different option and we could just print out in the following format:

# <name>:<sp><desc>
train: Train *model*
evaluate: Evaluate *model*

And, we can easily provide all of the stage internals through --show-json. I have no issues with that.

Alternately, we could make repro/exp return a list of the stages, as a help command for the stages.

[shcheklein]

dag is more of a visual command

but it's not like it has to be, right? also, we can make --dag option. It just feels that we have to much overlap, and it's hard to justify the whole top level command for this.

Could that be the other way around? Encourage people to use --short/-s for parsing/grepping?

I don't know. It's my personal bias I guess from regular CLI tools, like ls. It still feels we can make a bit more compact. Also if we can make it parsable by default it's better to my mind. We should have strong reasons to not do so.

ls might not work as the columns might vary based on description/outputs/dependent nodes/dependencies

skip values? not sure, would need to see the whole table schema I guess

-s should be machine-readable.

agreed, if we can make regular output grep/awk friendly - better do that. But --show-json I mention for consistency, an interface we guarantee consistenct?

And, we can easily provide all of the stage internals through --show-json. I have no issues with that.

👍

Alternately, we could make repro/exp return a list of the stages, as a help command for the stages.

make sense, might be an option as well

[jorgeorpinel]

Woa, interesting!

• I agree that the centering is unexpected (feel like a retro DOS app which is fun but probably too much?)
• Agree about -s default and more compact details
• Agree that dag is a separate thing (as long as it makes sense that this command doesn't validates DAGs).
• Cool md support but maybe add a flag to enable that (otherwise truncate desc heavily).
• ➕1 on all stage details via --show-json (if requested)
• dvc repro list feels like something repro shouldn't worry about.

The documented feature of this is the autocompletion

What docs changes does that entail? Thanks

[jorgeorpinel]

the list-layout is experimental

On this. I'm not sure about listing deps and outs (or even more details) at least initially (until someone asks for it?). You could cat dvc.yaml for that (mentioned in #3743 (comment)). We "sell" dvc.yaml as human-friendly so I think we expect users to be OK with eye-parsing simple YAML.

[skshetry]

• I agree that the centering is unexpected (feel like a retro DOS app which is fun but probably too much?)

UI is no longer centered. The demo is a bit old. Updated with the screenshot.

[skshetry]

We "sell" dvc.yaml as human-friendly so I think we expect users to be OK with eye-parsing simple YAML.

Tell this to a user with 100 stages/~1300 lines of dvc.yaml.

It's human-friendly in terms of readability and writability and that too applies only to a certain extent; it does not enlighten the user as to their relationship/purpose just by looking at it.

[jorgeorpinel]

a user with 100 stages/~1300 lines of dvc.yaml

Is there a real life example of this out there, and most importantly has it been reported/requested by a user with such huge dvc.yaml files? In any case how does this command help with that? The output is even longer than the DVC files.

It's human-friendly in terms of readability

➕

it does not enlighten the user as to their relationship/purpose just by looking at it

OK but again, how does this command help with that? It basically reformats the same info from dvc.yaml. It doesn't make any "enlightening" interpretations.

[jorgeorpinel]

As a reminder, I'm not arguing against the command in general. I'm questioning the value of showing all the details from dvc.yaml (except in --show-json, as a programmatic interface to out dvc.yaml format) until we have evidence of this need from users. But if that exists and I'm not aware of it, maybe you can help us by sharing the full context?

p.s. the purpose in #3743 doesn't seem to require full stage details IMO, including the dvc.yaml location seems enough.

[jorgeorpinel]

p.p.s. any details that can fit in a very compact manner (one line) I wouldn't see a problem with — if the output remains easily readable.