Use of subcommands in the CLI syntax

[lassik]

I'm late with these comments, but anyway, the issue of subcommands needs some long-term thinking. On one hand, it's nice to be able to do just unibeautify file.c. On the other hand, this makes the unibeautify command's syntax harder to extend in the future.

The current unibeautify command supports giving the filename as unibeautify foo, but also supports the subcommand unibeautify support. I would strongly encourage us not to mix these two usages. Shell commands should be easy to call from shell scripts and other programs, so it's customary to do things like unibeautify $file, where the value of the variable $file can basically come from anywhere. Usually this is protected as unibeautify -- $file so that $file is not interpreted as a command line option if it happens to start with a dash (the -- argument disables option parsing for the rest of the command line, any options that come after it are treated as ordinary arguments even if they start with a dash).

Anyway, the overarching principle is that any command line argument can come from a variable, so an argument at a particular position should always have only one kind of meaning. So we should have either:

1. unibeautify $file and unibeautify --supports, or:
2. unibeautify beautify $file and unibeautify supports

For the same reason, command line flags should not take a variable number of arguments. I don't think we have any of these :)

Of course, the subcommand names are up for debate :)

[lassik]

Hmm. I could think about this over the weekend and try to sketch a subcommand-based syntax that's easy to script reliably yet not too verbose for common tasks. It's not trivial to balance those conflicting needs but maybe a decent solution can be found. I have lots of experience dealing with the pain points of various command line tools and the nitpicky personality to do it 😄

[Glavin001]

I have lots of experience dealing with the pain points of various command line tools and the nitpicky personality to do it

Awesome! We definitely need that 😄 . Your input is very much appreciated.

I have not thought about the CLI much lately -- have been working on website/documentation and Unibeautify CI preparation. Looking forward to hearing more of your ideas!

[lassik]

Thanks :) I'll try to sketch it up. In a way, it's good that Emacs package has to rely entirely on the CLI, since developing those in tandem ensures that the CLI ends up actually being easy and reliable to script. I think a Vim plugin would also have to be done completely via the CLI (and probably most other editors that don't have native JavaScript). If you can recruit someone to do a Vim plugin, we would have even more assurance that the CLI is done right 😄

[lassik]

Late with this, but finally arrived at a solution that I personally think is OK:

unibeautify beautify [<source-file> ...]

unibeautify validate source [<source-file> ...]
unibeautify validate config-file <config-file>
unibeautify validate config-json <json-string>

unibeautify property has-config <source-file>
unibeautify property config-file <source-file>
unibeautify property config-json <source-file>
unibeautify property project-root <source-file>

unibeautify supports
unibeautify supports language [<language>]
unibeautify supports beautifier [<beautifier>]

• The beautify subcommand would be used to beautify files (or stdin when no filename is given).
• validate source would be used to check whether source code is correctly beautified (e.g. for commit hooks or CI jobs, gofmt has a feature like this).
• validate config* are used to validate Unibeautify configurations, again for CI jobs or troubleshooting.
• The property subcommand is a generic query interface for file metadata, mainly for scripting, and easy to extend with new queries as needed.
• supports is meant for querying things about the Unibeautify implementation itself (instead of source files, config files or other "user" data).

It might be reasonable to combine property and supports into one subcommand but when I look at that overall syntax, I personally find the current separation natural. property may not be the best name for that subcommand, and the other names are up for debate too.

This syntax is easy to extend, easy to script reliably and ought to be pretty easy to comprehend at first sight. In my opinion, it's also aesthetically good enough (but I'm quite heavily biased toward reliability and simplicity at the expense of brevity and shortcuts). Some might find it too verbose.

The biggest problem here is that one can't simply do unibeautify foo.js to beautify a file. Instead we have to do unibeautify beautify foo.js. I would suggest solving this by having a shorthand for the beautify subcommand, e.g. the letter b. The usage unibeautify b *.js doesn't look terrible to me, considering the benefits of having a very regular syntax. If we permit unibeautify foo.js then the syntax for all the other subcommands will have to be complex and iffy - and probably also harder for users to understand, especially all the edge cases.

Most of these subcommands would take flags (filenames, output formats, quiet/verbose, etc.) but I thought it'd be best to agree on the basic command set before getting into those details.

[lassik]

Actually I just realized that the filename arguments should probably not be optional. If the list of source files comes from an array that happens to be empty, then unibeautify will try to beautify from stdin instead of beautifying files - not what the programmer intended. E.g. in Python:

sourcefiles = glob.glob("*.c") # What if the glob doesn't return any .c files?
subprocess.check_call(["unibeautify", "beautify", "--"] + sourcefiles)

Generally, all kinds of optional arguments tend to be a bad idea in a CLI syntax, for similar reasons :D The purpose of --flags is to supply optional arguments, everything else should be required.

[lassik]

Here are some examples of how one might script this interface, in Unix shell syntax:

if unibeautify supports language C; then
    unibeautify beautify --inplace -- *.c
fi

if unibeautify property has-config -- "$sourcefile"; then
    echo "The Unibeautify settings for the source file are:"
    unibeautify property config-json -- "$sourcefile"
else
    echo "No corresponding .unibeautifyrc* found for this source file"
fi

echo "The following files have NOT been beautified:"
git ls-files | xargs unibeautify validate source --
# This usage doesn't look intuitive, so maybe "validate" is not a good name.
# The idea is that `unibeautify validate` prints the names of all files that need
# attention (for processing, reporting, etc.). If everything is fine then it's silent.

[lassik]

For comparison, if we lumped all the subcommands together instead of dividing them in groups, it would look something like this:

unibeautify beautify [<source-file> ...]
unibeautify validate-source [<source-file> ...]
unibeautify validate-config-file <config-file>
unibeautify validate-config-json <json-string>
unibeautify source-file-has-config <source-file>
unibeautify source-file-config-file <source-file>
unibeautify source-file-config-json <source-file>
unibeautify source-file-project-root <source-file>
unibeautify list-supported-languages
unibeautify list-supported-beautifiers
unibeautify supports-language <language>
unibeautify supports-beautifier <beautifier>

Still reliable and extensible, but IMHO not as elegant, and maybe not as easy to understand at a glance.

[Glavin001]

@lassik This looks great! 🎉

if we lumped all the subcommands together instead of dividing them in groups,

I like the groups as described in #82 (comment) 👍 .

It might be reasonable to combine property and supports into one subcommand but when I look at that overall syntax, I personally find the current separation natural. property may not be the best name for that subcommand, and the other names are up for debate too.

Could you explain a little more what property does?

if unibeautify property has-config -- "$sourcefile"; then
    echo "The Unibeautify settings for the source file are:"
    unibeautify property config-json -- "$sourcefile"
else
    echo "No corresponding .unibeautifyrc* found for this source file"
fi

Looks like it checks for metadata specifically involving the sourcefile, where as the supports group would be to check Unibeautify's support in general (e.g. can beautify language, etc)?

The biggest problem here is that one can't simply do unibeautify foo.js to beautify a file. ... If we permit unibeautify foo.js then the syntax for all the other subcommands will have to be complex and iffy - and probably also harder for users to understand, especially all the edge cases.

I am OK with unibeautify beautify foo.js 😃 .

Actually I just realized that the filename arguments should probably not be optional.

Your case there makes sense.

In general, Unibeautify treats the filename as being optional. This is valuable in cases such as beautifying an unsaved text editor.

[lassik]

Could you explain a little more what property does?

Looks like it checks for metadata specifically involving the sourcefile, where as the supports group would be to check Unibeautify's support in general (e.g. can beautify language, etc)?

Yes, exactly. Quoted from above:

supports is meant for querying things about the Unibeautify implementation itself (instead of source files, config files or other "user" data).

It might be reasonable to combine property and supports into one subcommand but when I look at that overall syntax, I personally find the current separation natural. property may not be the best name for that subcommand, and the other names are up for debate too.

I am OK with unibeautify beautify foo.js.

That's awesome 🎉 We can also provide tab completion for bash/zsh. I can look into that.

In general, Unibeautify treats the filename as being optional. This is valuable in cases such as beautifying an unsaved text editor.

Agreed.

[lassik]

Trying to work out the details of the beautify subcommand. This is hard! 😄

This is the most elegant thing I could manage:

unibeautify beautify diff [-o out.diff] --stdin
unibeautify beautify diff [-o out.diff] file.js
unibeautify beautify diff [-o out.diff] *.js

unibeautify beautify cat [-o out.js] --stdin
unibeautify beautify cat [-o out.js] file.js
unibeautify beautify cat [-o out.js] *.js

unibeautify beautify rewrite file.js
unibeautify beautify rewrite *.js

So diff and cat would be by analogy to those popular Unix commands. Unfortunately it adds another layer of subcommands (diff|cat|rewrite).

All of these would take the following common flags (maybe add one-letter abbreviations):

• --config-file <filename> (else lookup .unibeautifyrc* based on input filename)
• --config-json '{...}'
• --error-file <filename> (default to stderr)
• --error-format <plain|ansi|json> (default to plain or ansi based on stderr)
• --language <language-id>
• --q/--v (quiet/verbose, -v can be given more than once for extra detail)

I can explain in painful detail why all of this is the way it is :D But how do you think it looks at face value? I think the main problem now is that with 3 levels the commands are so long.

One thing that could be simplified about this is that diff and cat have the exact same syntax (flags and arguments) so they could be combined into one subcommand. We'd just add an extra flag to specify the output format (code vs diff). But when I wrote out that command set on the screen, it just didn't look as intuitive as the one above. I think that's because humans intuitively think of diff and cat as completely different operations, even though they are not that different in a formal sense.

[lassik]

Here's a variant that replaces the word cat with the more obvious show. We can do that since I removed the multi-file version (unibeautify beautify cat *.js). That's a bit of a questionable command to have. (Beautify multiple input files into a single output file - what does that mean? Package them up into a tar/zip or json format? Concatenate them? The command cat made it clear that they would be concatenated. But is there any non-contrived scenario where that is a useful thing to do? I imagine it would be more likely to confuse people.)

unibeautify beautify diff [-o out.diff] --stdin
unibeautify beautify diff [-o out.diff] file.js
unibeautify beautify diff [-o out.diff] *.js

unibeautify beautify show [-o out.js] --stdin
unibeautify beautify show [-o out.js] file.js

unibeautify beautify rewrite file.js
unibeautify beautify rewrite *.js

Actually I now prefer this because it's even more obvious. And having diff and show as different subcommands is now justified because diff has the multi-file syntax form that show does not have :p

[lassik]

Since diff, show and rewrite are the most essential subcommands, we could also remove the beautify prefix altogether, and have them directly as top-level commands:

unibeautify diff [-o out.diff] --stdin
unibeautify diff [-o out.diff] file.js
unibeautify diff [-o out.diff] *.js

unibeautify show [-o out.js] --stdin
unibeautify show [-o out.js] file.js

unibeautify rewrite file.js
unibeautify rewrite *.js

unibeautify validate source
unibeautify validate config-file
unibeautify validate config-json

unibeautify property has-config
unibeautify property config-file
unibeautify property config-json
unibeautify property project-root

unibeautify supports
unibeautify supports language
unibeautify supports beautifier

This now gets my vote 😄

diff and show are analogous to git diff and git show. But the word show is still non-ideal IMHO.

[lassik]

Here's yet another attempt 😄 Thought more about the diff command...

So the main point of unibeautify is to transform code into beautified code. Diffs are a side product of this process. We can beautify code without making diffs, but we can't make diffs without beautifying code. So whatever kind of output we want, it always involves beautifying code.

And diffs are just one kind of side product. We can produce any number of other kinds of side products: JSON reports, HTML reports, XML reports, lists of filenames - even the exit code of unibeautify is one side product. And we can produce any number of side products in parallel.

Since beautified code is the primary product, we could structure the commands around where it goes:

unibeautify rewrite  [--diff out.diff] [--html out.html]  file.js
unibeautify rewrite  [--diff out.diff] [--html out.html]  *.js

unibeautify show     [--diff out.diff] [--html out.html]  [-o out.js] --stdin
unibeautify show     [--diff out.diff] [--html out.html]  [-o out.js] file.js

unibeautify crunch   [--diff out.diff] [--html out.html]  --stdin
unibeautify crunch   [--diff out.diff] [--html out.html]  file.js
unibeautify crunch   [--diff out.diff] [--html out.html]  *.js

• rewrite - code goes right back into the files it came from
• show - code goes to stdout or to the specified output file
• crunch - code goes into a black hole - we are only interested in side products

All of these commands would have the same flags for generating side products:

• --diff - generate a unified diff
• --html - some kind of human-readable report for CI
• --json - machine parseable report
• --file-list - list of filenames processed, one per line
• --tar - tar archive containing the beautified files
• sky's the limit...

Each flag would take a filename argument saying where the output goes. Since crunch and rewrite do not write anything to stdout, the filename '-' could be used to write one of the side products to stdout (e.g. unibeautify crunch --diff - to write a diff to stdout)

[lassik]

Even more 😄 After playing around with it, I think stat is more natural than crunch.

unibeautify show [-s ...] [-e ...] [-o out.js] --stdin
unibeautify show [-s ...] [-e ...] [-o out.js] file.js

unibeautify stat [-s ...] [-e ...] --stdin
unibeautify stat [-s ...] [-e ...] file.js
unibeautify stat [-s ...] [-e ...] *.js

unibeautify rewrite [-s ...] [-e ...] file.js
unibeautify rewrite [-s ...] [-e ...] *.js

So all these commands have identical -s options to produce statistics/side-product files.

It would work like this to write some stats to stdout:

unibeautify stat -s diff *.js
unibeautify stat -s html *.js
unibeautify stat -s json *.js
unibeautify stat -s list-beautiful-files *.js
unibeautify stat -s list-ugly-files *.js

You could do the same with rewrite. But it would be an error with show to try -s into stdout.

You could redirect stats to a file:

unibeautify stat -s diff:foo.diff *.js
unibeautify stat -s json:/path/to/foo.json *.js
# etc.

Maybe to a file descriptor:

unibeautify stat -s diff%3 *.js
unibeautify stat -s json%3 *.js
# etc.

The -e option is similar to -s, and controls error output (default is plaintext/ansi to stderr). Whereas -s outputs to stdout by default if an output filename/descriptor is not given, -e outputs to stderr by default. Also, -e has different output filters, e.g. -e json outputs different data than -s json.

unibeautify rewrite -e json *.js  # json errors to stderr
unibeautify rewrite -e json%1 *.js  # json errors to stdout
unibeautify rewrite -e xml:errors.xml *.js  # xml errors to errors.xml
unibeautify rewrite -e none *.js  # don't write any errors anywhere (think about -q flag)

# This would write errors to XML file, JSON file, *and* normal plaintext/ansi stderr:
unibeautify rewrite -e xml:errors.xml -e json:errors.json -e '%2' *.js

# This combines -s and -e flags:
unibeautify show -s diff:out.diff -s json:report.json -e json:err.json --stdin

So the applications can get quite bewildering in the end when many of these flags are combined, but the important thing is that the option scheme is still very simple and symmetrical. Should be easy to implement and use safely. And the common use cases also look very simple.

[lassik]

Okay, check this out, this is the simplest yet:

unibeautify filter  [-o ...] [-e ...] --stdin
unibeautify filter  [-o ...] [-e ...] *.js
unibeautify rewrite [-o ...] [-e ...] *.js

unibeautify property has-config
unibeautify property config-file
unibeautify property config-json
unibeautify property project-root

unibeautify supports
unibeautify supports language
unibeautify supports beautifier

So stat (nee crunch) and show (nee cat) are now combined into a single command named filter. filter and rewrite have the exact same syntax.

The -e flag works as above, writing errors into files or file descriptors.

The -o flag combines the old -o and -s flags. Beautified code is now just another output format (we could name it simply code).

So you'd use it like this:

 # rewrite your .c files
unibeautify rewrite *.c

# rewrite your .c files and also show a diff on stdout
unibeautify rewrite -o diff *.c

# beautify a file, store the new code in a different file
unibeautify filter foo.js -o code:newfoo.js

# beautify stdin to stdout (produce the diff)
unibeautify filter -o diff --stdin

# beautify stdin to stdout (produce the beautified code)
unibeautify filter -o code --stdin

# same, but also write json errors to stderr and a diff to file descriptor 3
unibeautify filter -o code -o diff%3 -e json --stdin

# this is kind of silly but can be done
unibeautify rewrite -o code file.js

# this is an error, because `-o code` can't be used with multiple input files
unibeautify filter -o code a.js b.js

# this is also an error because we have `-o code` and multiple input files
unibeautify rewrite -o code a.js b.js

# this would be a no-op, since no inputs and no outputs are given
unibeautify filter

# these would do nothing except write errors to stderr and exit zero/nonzero
unibeautify filter file.c
unibeautify filter --stdin

I think this is about as simple, natural and extensible as we can make it.

[lassik]

The file descriptor support seemed weird at first but it's starting to grow on me. Tempfiles are the bane of inter-process communication. It's much cleaner to pass everything around with pipes. If we support routing output to arbitrary file descriptors, callers can create a pipe for every kind of output they need, then just assign the pipes to pre-determined file descriptors and tell unibeautify about those fd numbers. No tempfiles ever 🎉

[lassik]

On safety

Because we use prefix syntax in the -o and -e flags (the output format comes before the file name), it's also easy to script safely:

unibeautify filter -e "json:$errfile"     # $errfile can be anything
unibeautify filter -o "diff%$descriptor"  # $descriptor can be anything

And since filter and rewrite are different commands, the user can be confident that unless the command starts with unibeautify rewrite, it will never rewrite their files.

The commands work properly when sourcefile names come from an array (they don't do anything weird for zero-length arrays):

# Filenames often come from $@ in bash.
unibeautify filter -o diff -- "$@"
unibeautify rewrite -- "$@"
# etc .

[lassik]

Normally Unix commands have a version subcommand or --version option. Instead of supports commands, we could have them as subcommands of version. This would reduce the number of words in the CLI syntax.

# Print human-readable version information in plain text
unibeautify version

# -o could be used similar to the filter and rewrite commands
unibeautify version -o json
unibeautify version -o json:ver.json -o xml:ver.xml

# list supported languages
unibeautify version languages
unibeautify version languages -o json:languages.json
unibeautify version languages C C++

# list supported beautifiers
unibeautify version beautifiers
unibeautify version beautifiers -o json:beautifiers.json

[lassik]

@Glavin001 General impressions so far? I have some inspiration so I could make a feature branch and try coding a first draft in about a week if you're on board with the general direction the design is going. Mainly I'd like to try out how intuitive it feels to use the filter and rewrite commands as above.

[Glavin001]

No tempfiles ever 🎉

• Use of subcommands in the CLI syntax #82 (comment)

@lassik A goal I share. I'd be open to receiving a Pull Request implementing this. Of course, we need to continue support for Linux, Mac, and Windows.

---

@lassik I think you have a great understanding of what needs to be done and a strong future-proof design. Could you update the Issue body with a summary of your final proposal? And use Task Lists so we can track progress: https://help.github.com/articles/about-task-lists/
🎉

[lassik]

@lassik I think you have a great understanding of what needs to be done and a strong future-proof design. Could you update the Issue body with a summary of your final proposal? And use Task Lists so we can track progress: https://help.github.com/articles/about-task-lists/
🎉

Thanks for entrusting this :) I'll try to get a working prototype in an experimental branch soon.

As for the proposal and task list, I think that depends heavily on how the design works out. It's almost impossible to tell how good an interface is without building prototypes, trying actual use cases and gathering feedback, as there are often unexpected edge cases lurking that can make a whole design unsound, and then the "gut factor" - a design that looks good on paper may not be good to use and vice versa. Honestly, the criterion I always use is, "keep trying things until it's pleasant to use" 😄

The sketches in this issue cover a wide range of use cases, so that gives some confidence that the overall design is workable, but by intuitive guess, 1-2 big changes are needed still.

[muuvmuuv]

I think since Unibeautify is a beautifier I would not add the beautification function as a subcommand. This would take users to type even more to beautify a file. I think we should keep this as simple as we can. So when the user is typing unibeautify filename.js it should simple beautify this file, everything else should be flags like --dry to only show results not write them or --language to use another language.

[lassik]

I agree that it would be simplest if people could just type unibeautify *.js to rewrite files. But we need to have subcommands like property, supports etc. Adding them all as flags (--property, --supports, etc.) would make the overall syntax extremely complex, only to make the normal case unibeautify *.js as simple as possible.

Some things we could do to help:

1. Let people say in .unibeautifyrc what files should be beautified (e.g. *.js). Then people could just type unibeautify rewrite (without any file names) to beautify all files in the project.

2. Have an abbreviation, so instead of unibeautify rewrite people could type unibeautify rw.

3. Instead of separate subcommands, have separate top-level shell commands: instead of unibeautify property and unibeautify supports, one would type unibeautify-property and unibeautify-supports (with the dashes). The main unibeautify command would only rewrite files, the other commands would do everything else.

Personally, I like options 1 and 2. Option 3 seems a bit too complex to me (what if installation is messed up so some of the commands point to a different version of Unibeautify than the others?)

[lassik]

The other question: Should the unibeautify command be destructive (rewrite files in-place) by default (instead of only showing the changes by default, i.e. dry run)?

Many things can possibly go wrong when beautifying a file, so the user should always have a backup of the file. Usually people have the file in version control (Git) which is a good backup. But we'd need some way to make sure of that before we overwrite their files. In my opinion, the default operation should not be destructive unless we have backups.

We could do this by:

1. Requiring the user to give a command or flag (e.g. unibeautify rewrite or unibeautify -w) to give permission to rewrite their files.

2. Check for known version control directories (.git, .hg, .svn, .cvs etc.). If we find one, we assume that the user has backups in version control.

3. If we don't find any of those directories, then make a backup file for each file we rewrite. E.g. if the beautifies makes changes to foo.js then it would save the old contents in foo.js.bak.

Unfortunately, if we don't do any of the above, it's likely that there will be some bug or problem and someone will complain that we deleted their code or messed up their files without warning ☹️

[lassik]

Using a Unibeautify editor plugin (VScode, Vim, Emacs, etc.) is great because the editor's own "undo" feature provides an automatic backup. But I agree that calling unibeautfy from the shell should also do something reasonable.

[muuvmuuv]

@lassik Is it not possible to have both? Using unibeautify *.js to beautify a file and unibeautify validate... as a subcommand.

I agree that it makes more sense to have it the other way around, so the user choose by its own to rewrite the file and I like your idea (2.) to look for version control. (maybe something we can add to global unibeautify options, for those who are sure about what they are doing)

[lassik]

@lassik Is it not possible to have both? Using unibeautify *.js to beautify a file and unibeautify validate... as a subcommand.

We actually have this in the current CLI syntax, but it's not customary to do that. The reason is that command line arguments should not have "magic" values that do something different from all the other possible arguments at that position on the command line. E.g. unibeautify supports.js would beautify that file but unibeautify supports (almost the same text) would do something completely different because supports is a magic value.

Often command arguments come from a variable in a shell script, e.g. unibeautify rewrite "$filename". In this case the script writer should be confident that the command will try to beautify a file, no matter what the value of $filename is. We could have some rule that e.g. if the filename contains a dot, then it's a filename, else it's a subcommand. But any such rule adds complexity for script writers to worry about.

The traditional Unix convention is that flags that start with a dash (-x or --whatever) are the only "magic" command line arguments. People can turn off flag processing with --, e.g. unibeautify rewrite -- "$filename", to make sure that $filename is never interpreted as a flag, even if it starts with a dash. The -- should always be used in shell scripts but unfortunately many people don't know about it. Most programs support it. I would strongly favor that we follow that convention, as it's simple, reliable and well known.

[lassik]

I don't know if some programs also turn off subcommand processing after --. So one could write unibeautify -- "$filename" and it would try to beautify $filename, even if $filename contains a magic value like supports. In any case, that's not a standard well-known convention.