docs: add examples for CLI reference

[aguschin]

While removing examples from CLI help in iterative/mlem#408, it was found that:

1. These commands didn't existed in docs before, so we need to take their examples and put them to docs: checkenv, config
2. deployment didn't have examples
3. Some commands in docs lacks the execution results, like this one:

   $ mlem types
   ...
   

@mike0sv, I wonder, should we also code examples in your bootstrap script so they + their output would be generated?

[jorgeorpinel]

@aguschin this ticket's title seems really broad. Do you refer specifically to checkenv, config, deployment, and types? Keep in mind we will never cover every scenario with examples so maybe not even all commands will have examples beyond basic usage (Idk).

Also, are you suggesting to copy examples from docstrings or help output? That would seem unnecessary to me: if there are examples in those places they should have a different purpose than the ones in mlem.ai/doc (moving them over to docs could make sense though).

[aguschin]

Also, are you suggesting to copy examples from docstrings or help output?

There were some examples in CLI --help that we initially wanted to copy in docs. Not all of them should be copied maybe, but some for sure. Do you have a formulated criteria in mind for this?

[jorgeorpinel]

if there are examples in those places (help output and/or docstrings) they should have a different purpose (vs. cmd ref online)

Do you have a formulated criteria in mind for this?

It's a good question. Thoughs:

Help output (if extended with examples) is like a man page: meant for people who don't want to leave the terminal and have enough knowledge of the tool to be able to stay there. Think of them: what kinds of problems will they want to solve 80% of the time?

Cmd refs are similar in that they're self-contained (should not require reading other parts of the docs once the user has working experience) but a) the format is different (less constrained, nicer, better maintained?) and the type of user who prefers opening https://mlem.ai/doc/command-reference is probably different (role-wise) or not yet as skilled as terminal maxis. Think of them.

Also, cmd ref examples can help expand on the description of features that are hard to describe (easier to demonstrate).

Sorry, it's not a full or complete criteria, and maybe there's overlap indeed -- but I'd err on the side of a single-source of truth for examples on x topic.

In the future, how can we test or measure this? Ideally, support cases, telemetry, and website analytics should be able to give us an idea but it may be too much effort rn. Maybe we can ask the DVC/ CML teams whether they have insights around this since it affects all CLI tools, really. Cc @efiop @dberenbaum @DavidGOrtega @casperdcl

[casperdcl]

my tuppence (/CC @shcheklein):

• CLI --help
  • succinct
  • WELL MAINTAINED esp. for local CLI tools (MLEM, GTO, DVC)
  • includes URL to online.cmd/ref
• online.cmd/ref
  • strict superset of CLI --help
  • adds additional info (hyperlinks to external info, FAQs, examples, cross-ref to user guides, etc.)
  • WELL MAINTAINED esp. for non-local CLI tools (CML) or for very complex tools with lots of relevant "additional info" (DVC)

[jorgeorpinel]

WELL MAINTAINED

May be easier to maintain in docs.

strict superset of CLI --help

Mandatory repetition of every example? Doesn't sound ideal to me.

Maybe we should discuss this in a prod/docs call bc/ I don't think we have established many practices around this.

[casperdcl]

CLI --help

WELL MAINTAINED

May be easier to maintain in docs.

😱 maintenance difficulty is tangential. A CLI-first tool is useless without excellent --help (the 1st and often only thing users will try before giving up).

online.cmd/ref

strict superset of CLI --help

Mandatory repetition of every example?

Our current process easily handles this (devs open 2 PRs, one implementing+documenting on tool repo and one documenting-with-FAQs/edge-cases on cmd ref repo). If said devs want to, it can be (partially) automated (vis. MLEM). There should be zero maintenance burden on docs team here.

[aguschin]

Mandatory repetition of every example?

@casperdcl, could you please share an example here? I was assuming you mean that you have examples in CLI and on docs website. I thought about CML initially, but don't see any examples or man pages for cml:

$ cml comment --help                                                                                                                
cml comment

Manage comments

Commands:
  cml comment create <markdown file>  Create a comment
  cml comment update <markdown file>  Update a comment

Global Options:
  --log     Logging verbosity
          [string] [choices: "error", "warn", "info", "debug"] [default: "info"]
  --driver  Git provider where the repository is hosted
    [string] [choices: "github", "gitlab", "bitbucket"] [default: infer from the
                                                                    environment]
  --repo    Repository URL or slug[string] [default: infer from the environment]
  --token   Personal access token [string] [default: infer from the environment]
  --help    Show help                                                  [boolean]

$ man cml                                                                                                                            
No manual entry for cml

[casperdcl]

with reference to this, CLI --help should not have any (lengthy) examples. Examples should only be on online.cmd/ref. I assumed when @jorgeorpinel said "repetition of every example" in his reply he meant to say "repetition of every --option-flag description."

For further clarification, I think:

• the online.cmd/ref should have an automated check to ensure all --option-flags match the CLI
• the online.cmd/ref Descriptions of said flags however need to be a strict superset of the CLI's Descriptions of said flags... A couple more sentences, cross-refs/URLs, etc. And this should be manually maintained by the eng teams:

devs open 2 PRs, one implementing+documenting on tool repo and one documenting-with-FAQs/edge-cases on cmd ref repo

Meanwhile for man pages, I don't really see much point tbh. I kinda prefer CLI --help | less + /regex or just using the online ref. Maintaining proper man pages is far harder and not as featureful as an online ref.

toy example

$ foo --help
Subcommands:
  bar:  Make bars. Likely needs `foo bat --install`.
  bat:  Manage Chiroptera.

$ foo bar --help
Make bars. Likely needs `foo bat --install`.
Options:
  ...
Full reference & FAQs: <https://foo.cmd/ref/bar>.

$ curl --output - https://foo.cmd/ref | pandoc -f html -t markdown
## Subcommands
- **bar**: Make [bars](https://en.wikipedia.org/wiki/Metasyntactic_variable).
  Consider running `foo bat --install` first to avoid [feeding issues](https://foo.cmd/faqs#feeding).
  See [foo#615](https://github.com/iterative/foo/issues/615) for an automated fix.
  See also [barring bats](https://foo.cmd/use-cases/dingbats).
- **bat**: Manage [Chiroptera](https://en.wikipedia.org/wiki/Chiroptera).
  They make up 20% of mammal species.

/CC @shcheklein

[jorgeorpinel]

Agree in general.

the online.cmd/ref Descriptions of said flags however need to be a strict superset of the CLI's Descriptions

If by superset we mean include the exact same wording, that doesn't seem sritical to me, TBH. I guess it would be the easiest way to automate the checks though. No strong opinion