Shell Autocompletion

Related: How Interactive Shells Work

Threads that started this

• Survey: What shell completion scripts do you use? (September 2018) -- started by Andy, 36 comments on lobste.rs

• Subthread on Hacker News -- people brought up argcomplete for Python, clap-rs for Rust, Language Server Protocol

  • Subthread about completion invariant -- e.g. you should be able to "spam" TAB and come up with something reasonable

• Running Bash Completion Scripts with OSH

  • lobste.rs comments -- Xiao (author of Elvish) and I discussed the possibility of shared completions. Discussed the git use case. Continued into e-mail thread (will summarize on Zulip)
    • Using and Writing Completions in Elvish
  • Reddit comments -- zsh developer and author Oliver Kiddle (okapi-a) discussed the possibility of shared completions. Continued into e-mail thread (will summarize on Zulip). Goal: Get upstream authors to maintain completion scripts?
    • Bash auto-completion in Clang -- Oliver brought this up as an example of something limiting. Doesn't take advantage of zsh features?

• Release of OSH 0.6.pre5 -- A long list of the things I had to do to emulate bash! Not done yet.

More Threads

• CLUI: Building a Graphical Command Line (May 2021)
  • https://github.com/withfig/autocomplete -- Fig autocomplete specs.
  • https://murex.rocks/docs/commands/autocomplete.html -- Murex shell autocomplete

Design

• Proposals
  • Shellac Protocol Proposal (renamed from SHCOMP Protocol Proposal)
  • Shellac Protocol Proposal V2
    • Shellac Test Cases
  • CLAP Grammar Proposal
• Design Notes
  • Autocompletion Use Cases
  • Diversity in Command Line Syntax
  • Parsing Models Cheatsheet
  • Parsing Shell Quotes Requires Parsing the Whole Language
  • Startup Time Measurements
  • Flag Parsers in Various Languages
  • Projects Already Doing Something Like Shellac
  • Completion Chat November 2019
  • Implementations of Shell Autocompletion
• Andy Questions / Research
  • Can we reuse fish's man page parser? It's a Python script.
  • How is zsh's 7000-line git completion generated? Can we reuse some of that logic?

Things I've Learned about Shell Auto-Completion (So Far)

Here is a draft of an Oil blog post:

• bash and zsh "boiled the ocean" in parallel, i.e. developed completions for common and not-so-common Unix commands.
• zsh maintains many completions in-tree. Bash is at least two different projects.
  • although there is also the zsh-completions and oh-my-zsh projects.
• bash does not use its own parser to figure out what the completion context is.
  • readline parses the line with COMP_WORDBREAKS.
  • Not only that, but bash-completion disregards that, and reparses COMP_LINE. bash is parsed in bash.
• bash dynamically greps the output of 'ls' and other commands for flag completion.
• git is the biggest completion by a factor of 2 in both bash and zsh.
• Some commands partially handle their own completion logic.
  • npm
  • clang
  • git --listcmds
  • Ryan Prior used https://github.com/davetron5000/gli (a Ruby command line parser) for Conjur tools and got completion info from GLI data structures.
• Some libraries handle their own completion. See the list above.
• Grammars / pure-declarative approaches are probably good enough for 90% of completions, but not the (IMO) most important use cases like git.
• Grammar-based approaches have been tried in the past
  • mbrubeck/compleat on Github
  • https://github.com/grammy-org
• zsh has a nicer completion UI.
  • It gives you help for flags.
  • It doesn't constantly redraw the line like readline! (argh)
• Thomas Ballinger (contributor to the BPython shell) has a great blog about Terminals.
  • It's easier to draw stuff below the prompt than above the prompt.
• Emacs and xonsh already do something like Shellac!!! They shell out to bash.
  • Projects Already Doing Something Like Shellac

Big questions

• Should we "boil the ocean" 0, 1, or N times?

  • 0: emulate bash
  • 1: share autocompletion among shells
  • N: everybody develops their own

• What would a shared completion API / protocol / grammar / data format look like?

Forum

• Visit https://oilshell.zulipchat.com/ and sign in with GIthub! _ Advantages:
  • Write async messages like e-mail, or chat synchronously in real-time.
  • Organize the conversation into threads (the git use case, grammars, Elvish design, etc.)
  • Markdown support for code examples
  • Notify via e-mail with @, but don't spam people's inbox with every message.
  • Public, so new people can view the history (unlike e-mail / IRC)
  • Easy login with Github.

Declarative/DSL/Grammar vs. Invoking Arbitrary Code Behind a Completion Server Protocol

• Power
  • Is the DSL Turing complete? Does it need to be?
• Latency
  • Something in-process has better latency.
• Version skew of completion definitions:
  • Querying a binary directly eliminates version skew. The upstream author adds a flag via argparse or golang/flags, and everything should work.
  • Otherwise there is an extra step of generating a grammar, and this can get out of date.
    • (There are many different ways to distribute binaries now. Package managers are only one. For example, the Rust toolchain is updated with a curl | sh-style shell script.)
• Version skew of the DSL itself.
  • Is it statically linked into every library?
• Killer app: Could you implement the completion server protocol in VimScript? Yes, as long as it can invoke an arbitrary process.
  • You probably cannot implement a DSL in VimScript, or it would be very awkward.