UI/UX of the stage:list command

[skshetry]

We introduced dvc stage list command for 2 reasons:

1. For autocompletion, and
2. To reduce the effort to look into dvc.yaml file each time.

Both of which are mentioned as a pain point in #3743.
I initially implemented it on #5191 as a POC with a rich UI and a format that can accommodate more information, but removed it later to the following format:

$ dvc stage list
prepare    Produces data/prepared
featurize  Produces data/features
train      Produces model.pkl
evaluate   Generates scores.json, prc.json

As the above format is one-liner, it is hard to add any more information.

There have been a few questions that are up for debate:

1. What should the format of the UI be? Should it be long-format or short by default? Should it be greppable/machine-readable by default?
2. Do we need more information here? If so, how should we accommodate it here, and in what format?
3. Is autogenerated description important? How should they be auto-generated? What things should be there?
4. Should we respect the desc field (right now it does) in the dvc.yaml file?
   etc.

---

It's a bit hard to express the full context around it as everyone has there contradictory view to it and what they expect here.
Please ask if there's anything regarding implementation that needs to be clarified, as most of the implementations right now are up for debate/questionable, so I am leaving it for clarity.

cc @dberenbaum @shcheklein @dmpetrov

[jorgeorpinel]

IMO:

1. Short by default and greppable (isn't it already since it's used for dvc shell completion?)
2. I liked all the info that came in the original POC. I thought this was gonna have a --show-json output to dump the entire object as parsed from dvc.yaml (once interpreted) — fully machine-readable?
3. Good Q! In order to keep within one line, maybe the best approach is a summary e.g. 1 cmd, 2 deps, 1 cmd, 1 out, 1 dep, 4 params, 2 cmds, 2 outs, 1 metric.
4. Idk about desc. What was it's original purpose of that field again? Could be long. Maybe only if you use --all or --desc flags.

p.s. for the record, a few other UI text enhancement suggestions can be found in #5282 (review).

[dberenbaum]

Sorry for being slow to respond here. What you have looks great to me. I have been hesitant to respond to the questions since I don't think adding another opinion will be that productive.

What do you think about using this opportunity to outline some standards for what to expect for output generally? Hopefully, that might save time in the future with needing to gather UI feedback on every command, and the UI will be more consistent.

This seems like one of many cases where the output is basically a table. There are at least 3 different output formats that I've seen for different dvc tables:

1. Fully machine readable (for example, json)
2. Pretty or at least focused on being human-readable
3. Hybrid with a focus on piping to unix utilities like awk, grep, etc.

Questions on these:

• Should all dvc commands with table-like outputs support all of these scenarios?
• Should the hybrid approach be its own format (for example, using --plain or --pretty to determine how human-readable to make the output)?
• What should be the default format?
• Can the default change depending on the use case, even if this create less consistency?
• What machine-readable formats should be supported (json, csv, tsv, yaml)?
• What should be prioritized in terms of unix piping (single-line entries, delimiters, ascii-only, etc.)?
• What should be prioritized in terms of human readability (colors, icons, headers, paging, length/width considerations, etc.)?
• What should be the options/flags to specify different formats?
• Is there some GUI version of this output in Viewer, VSCode, etc., and how does that impact the CLI formatting (consistency with the GUI, need to replicate GUI features, priority of human readability in CLI)
• What about an HTML format?

Feel free to push back if any of this is too broad or not useful. I can also open a separate issue to not block this specific use case.

[jorgeorpinel]

Hybrid with a focus on piping, with --show-json for fully machine-readable IMO.

I'd definitely extract all the secondary/generalized questions to a separate issue, if we're trying to decide on stage list soon at least.

[dberenbaum]

@skshetry I'm fine moving forward with the suggestions you and @jorgeorpinel have proposed here. As far as what to show for descriptions, we could potentially have the 2 deps, 1 cmd, 1 out field and an additional field for anything manually entered into desc, although we would probably want a character limit on what is shown there.

I'm not going to make a separate issue at the moment for the other questions I raised since we have enough other priorities, but I know you have been doing some research in these areas, so it would be great to hear anything you have found and any thoughts on UI design (doesn't have to be in this issue).

[dberenbaum]

To clarify above, it might be better to have separate fields for desc and the 2 deps, 1 cmd, 1 out field (summary?) so that users who include descriptions still have a way to see that.

[skshetry]

I should have introduced the --zsh and --bash flags for autocompletion instead. 😭
Though, I guess it's better to break it for once (only for zsh).

---

it might be better to have separate fields for desc and the 2 deps, 1 cmd, 1 out field

What should the order be? <name> <summary> <description>?
I don't think n cmd is important at all.

---

I still question why we need to make it machine-readable by default. 😄 With a longer form of output, we could provide more context here by default.

[dberenbaum]

What should the order be? <name> <summary> <description>?
I don't think n cmd is important at all.

👍

I still question why we need to make it machine-readable by default. 😄 With a longer form of output, we could provide more context here by default.

I liked the verbose pretty output you had before. No reason we can't still do that with a --pretty or similar flag. You could even start with that and then make a more succinct version for machine-readable.

Since more consistent UI is a goal, making machine-readable the default is probably a good idea since it might be important for other commands, and it keeps the output somewhat simple, which should make it easier to maintain consistency.

[shcheklein]

I would also distinguish machine-readable and cli tools readable (granted they might overlap). Machine readable in my head is about --json (it's not human readable at all), while making default output consumable by awk, grep, etc is a good practices that will benefit users.

[skshetry]

Closing as not planned.