Stan CLI user API

[mitzimorris]

proposal for new programmatic interface to core Stan algorithms

link to rendered proposal: https://github.com/stan-dev/design-docs/blob/stancli-user-api/designs/0034-stancli-api.md

[WardBrian]

This is very exciting!

A few general thoughts, right off the bat:

1. I tend to find it much easier to read/type command line arguments that --look-like-this (so called kebab-case) rather than --ones_with_underscores. In stanc3 we have deprecated any command line arguments with _ in favor of versions with - as the separator.
2. I understand the desire to avoid deeply nested and order-dependent argument parsing. I also think it is useful if we have exactly 1 level of nesting, e.g. the executable has subcommands (not prefixed with --) for very different purposes, similiar to tools like apt, git, etc. I think one level makes --help pages clearer and is useful for communicating what the different options are, rather than having a --help which lists both settings for HMC treedepth and how to get compiled model information (assuming something like ./mymodel info would be included).
3. "hmc", "optimize", "laplace", "variational", "eval-log-prob" seems like it is a mix of the names CmdStan currently has and of the specific algorithm names - is there a reason to not go full in and rename optimize as lbfgs and variational as advi?

And a couple questions:

1. Does cli11 support something like "take this JSON file and treat it as my config"? Especially for complicated configs it would be super useful. This is certainly optional though
2. Do you imagine this executable will be built in the same was as cmdstan executables (e.g. with make), or is that UI also getting another look?

[bob-carpenter]

I would like to see what the doc would look like for the command-line interface.

Question 1: Is there going to be an executable compiled from a model?

Question 2: Are there going to be subcommands corresponding to the operation?

Question 3: How are arguments specified?

I assume the answer to Q1 is yes, to Q2 is yes with one level of nesting, and that Q3 is up for grabs. I can imagine that Q1 is no and you build a higher-level driver that also compiles the model.

Then a spec should look something like this standard command-line doc. As an example, here's the head of the doc for git resulting from man git in the terminal.

GIT(1)                                          Git Manual                                         GIT(1)

NAME
       git - the stupid content tracker

SYNOPSIS
       git [-v | --version] [-h | --help] [-C <path>] [-c <name>=<value>]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|-P|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           [--super-prefix=<path>] [--config-env=<name>=<envvar>]
           <command> [<args>]

DESCRIPTION
       Git is a fast, scalable, distributed revision control system with an unusually rich command set
       that provides both high-level operations and full access to internals.

       See gittutorial(7) to get started, then see giteveryday(7) for a useful minimum set of commands.
       The Git User’s Manual[1] has a more in-depth introduction.

       After you mastered the basic concepts, you can come back to this page to learn what commands Git
       offers. You can learn more about individual Git commands with "git help command". gitcli(7) manual
       page gives you an overview of the command-line command syntax.

       A formatted and hyperlinked copy of the latest Git documentation can be viewed at
       https://git.github.io/htmldocs/git.html or https://git-scm.com/docs.

OPTIONS
       -v, --version
           Prints the Git suite version that the git program came from.

           This option is internally converted to git version ... and accepts the same options as the
           git-version(1) command. If --help is also given, it takes precedence over --version.

       -h, --help
           Prints the synopsis and a list of the most commonly used commands. If the option --all or -a
           is given then all available commands are printed. If a Git command is named this option will
           bring up the manual page for that command.

           Other options are available to control how the manual page is displayed. See git-help(1) for
           more information, because git --help ... is converted internally into git help ....

So for us, that'd be something like the following, where nuts_diag_adapt picks out what is now specified with method = sample, algorithm = hmc, metric = diag_e, algorithm engine = nuts, and adapt engaged = true.

<model.exe> nuts_diag_adapt
  --step_init <real+>
  --metric_init <file>
  --max_depth <nat>
  --warmup_iter <nat>
  --sampling_iter <nat>
  --save_warmup <bool>
  --thin <nat>
  --adapt_gamma <real+>
  --adapt_delta <prob>
  --adapt_kappa <real+>
  --adapt_init_buffer <nat>
  --adapt_term_buffer <nat>
  --adapt_window <nat>
  --sample_file = <file>
  --adapt_file = <file>
  ... ?

We also need the ranges (nat is for "natural", which is 0, 1, ...) and default values in the longer doc. And of course, I'm just guessing on things like sample file and adaptation file. You may also need a specification for output in ASCII, JSON, or binary format.

[mitzimorris]

1. I tend to find it much easier to read/type command line arguments that --look-like-this (so called kebab-case)

will use kebab-case

2. I understand the desire to avoid deeply nested and order-dependent argument parsing. I also think it is useful if we have exactly 1 level of nesting, e.g. the executable has subcommands (not prefixed with --) for very different purposes

3. "hmc", "optimize", "laplace", "variational", "eval-log-prob" seems like it is a mix of the names CmdStan currently has and of the specific algorithm names - is there a reason to not go full in and rename optimize as lbfgs and variational as advi?

excellent suggestions - will do

1. Does cli11 support something like "take this JSON file and treat it as my config"? Especially for complicated configs it would be super useful. This is certainly optional though

yes! https://github.com/CLIUtils/CLI11#configuration-file

2. Do you imagine this executable will be built in the same was as cmdstan executables (e.g. with make), or is that UI also getting another look?

the former, simply because I don't have enough imagination for the latter. open to suggestions.

[bob-carpenter]

command line arguments that --look-like-this (so called kebab-case)

I also prefer kebab case. The only issue is that the corresponding Python variable, look_like_this, has to be in so-called bruschetta case (I wanted to keep up the food metaphor for those who don't eat snakes).

[mitzimorris]

cf https://www.gnu.org/software/libc/manual/html_node/Argument-Syntax.html

"Option names are typically one to three words long, with hyphens to separate words."