Stablize Rust-Native Completion Engine Tracking Issue

[epage]

Maintainer's notes:

Remaining work for feature parity

• Communicate with bash (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete path ValueHints (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete possible values (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete short flags (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete long flags (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete subcommands (feat(complete): Skeleton for Rust-driven completions #3656)
• Communicate with zsh (zsh support for native completions #3916)
• Communicate with fish (fish support for native completions  #3917)
• Communicate with Powershell (powershell support for native completions  #3918)
• Communicate with Elvish (elvish support for native completions  #3919)
• Flag values (Support flags with values in native completions #3920)
• Multiple values (Support multiple values in native completions #3921)
• Hidden (Support hiding flags/subcommands/possible values/aliases for native compeletions #3951)
• Handling of space (Intentionally control when a space is appended for native completions #5587)
• Fix handling of OsString to preserve non-UTF8 paths
• Document that stdout should not be used before getting to completion processing in case the completions are requested (docs(complete): Expand dynamic docs #5643)
• FIGNORE support?
• Fulfill Document how to source completions #2488 for these new completions. Ideally, we have the registration script sourced on-demand in the docs so that it is "auto-updating" with the users tool
• Switch native completions to encourage env var initialization #3930
• Verify each shell's behavior with quoting to see if we need to add quoting to what we output
• clap_complete::env::Elvish generates code deprecated in 0.18, removed in 0.21 #5729
• Bash outputs everything as one completion, rather than a list of completions #5730

Non-blocking work

• Smarter selection of what to show (Filter and order completions based on input and importance for native completions #5058)
• Delimited values (Support delimited values for native completions #3922)
• is_require_equal_set support (Support is_require_equal_set in native completions #3923)
• last support
• trailing_var_arg support
• Support subcommand flags in native completion engine #5284 (doesn't appear supported by static completions)
• Removing options based on conflicts (self and others)
• Any App or Arg setting not currently handled by the static completions
  • Priority to value delimiter and requires equals
• Complete more ValueHints
• Figure out where this will live as the dependency requires are dramatically different than the static completions and this shouldn't be behind a feature flag for forever
• Custom help messages (Allow using a completion-specific candidate description over the candidate's help message in rust-native completions #5063)
• Completing multiple path elements at once (Complete multiple path elements at once #5279)
• nushell support (Dynamic completion for Nushell #5840)

Design considerations

• Make it easy to use for the natively supported shells but allow other shells to be supported

Prior art

• https://github.com/pacak/bpaf
• https://pypi.org/project/argcomplete
• https://github.com/rsteube/carapace
• https://github.com/microsoft/inshellisense
• https://github.com/sigoden/argc-completions
• https://click.palletsprojects.com/en/8.1.x/shell-completion/

---

#3022 was a tipping point for me in realizing that maybe our current approach to completions doesn't work. We effectively have to implement a mostly-untested parser within each shell. Examples of other problems that seem to stem from this:

• clap_generate fish: ArgEnum completions not working on toplevel command #2729
• Allow completion of partial commands [Powershell] #2750
• clap_generate PowerShell script does not complete case sensitive options like -S. Only completes -s #2145
• zsh: optional flag with takes_value breaks completions #1822
• Allow the completed command to be an alias #1764

If we take the approach of argcomplete where we do the parsing in our core code, rather than in each completion script, this will help us share parsing logic between shell, share some or all parsing logic with clap itself, and make a subset of the logic more testable.

We also need to decide whether to morph the existing parser into supporting this or create a custom parser (since the needs are pretty special). If we do a custom parser, we should probably do #2915 first so we can reuse lexing logic between clap and the completion generation. I've also been considering #2912 which would allow reusing the completion logic with any CLI parser.

[epage]

Bash's expectations

Inputs for -F and -C:

• COMP_LINE: The current command line. This variable is available only in shell functions and external commands invoked by the programmable completion facilities
• COMP_POINT: The index of the current cursor position relative to the beginning of the current command. If the current cursor position is at the end of the current command, the value of this variable is equal to ${#COMP_LINE}. This variable is available only in shell functions and external commands invoked by the programmable completion facilities
• COMP_KEY: The key (or final key of a key sequence) used to invoke the current completion function.
• COMP_TYPE: Set to an integer value corresponding to the type of completion attempted that caused a completion function to be called: TAB, for normal completion, ‘?’, for listing completions after successive tabs, ‘!’, for listing alternatives on partial word completion, ‘@’, to list completions if the word is not unmodified, or ‘%’, for menu completion. This variable is available only in shell functions and external commands invoked by the programmable completion facilities
• 1: name of the command whose arguments are being completed
• 2: word being completed
• 3: word preceding the word being completed on the current command line

Inputs for -F:

• COMP_WORDS, COMP_CWORD when used with -F

Output for -C:

• print a list of completions, one per line, to the standard output. Backslash may be used to escape a newline, if necessary.

Output for -F:

• It must put the possible completions in the COMPREPLY array variable, one per array element.

See

• https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html#Programmable-Completion
• https://www.gnu.org/software/bash/manual/html_node/A-Programmable-Completion-Example.html#A-Programmable-Completion-Example
• https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion-Builtins.html#Programmable-Completion-Builtins
• https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html

[epage]

argcomplete emulates bash's interface in fish by

    set -x COMP_LINE (commandline -p)
    set -x COMP_POINT (string length (commandline -cp))
    set -x COMP_TYPE

and then just registers that function

https://github.com/kislyuk/argcomplete/blob/develop/argcomplete/shell_integration.py#L60

[epage]

Value hints are supported on

• zsh
• fish

Tooltips are supported on

• zsh
• powershell
• elvish
• fish

We should make sure we don't lose these features as part of this transition, ie we shouldn't drop to the lowest common denominator.

[epage]

For Powershell

Register-ArgumentCompleter
        -CommandName <String[]>
        -ScriptBlock <ScriptBlock>
        [-Native]
        [<CommonParameters>]

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/register-argumentcompleter?view=powershell-7.2

The block receives

When you specify the Native parameter, the script block must take the following parameters in the specified order. The names of the parameters aren't important because PowerShell passes in the values by position.

• $wordToComplete (Position 0) - This parameter is set to value the user has provided before they pressed Tab. Your script block should use this value to determine tab completion values.
• $commandAst (Position 1) - This parameter is set to the Abstract Syntax Tree (AST) for the current input line. For more information, see Ast Class.
• $cursorPosition (Position 2) - This parameter is set to the position of the cursor when the user pressed Tab.

The block provides CompletionResult

The CompletionResult object allows you to provide additional details to each returned value:

• completionText (String) - The text to be used as the auto completion result. This is the value sent to the command.
• listItemText (String) - The text to be displayed in a list, such as when the user presses Ctrl+Space. This is used for display only and is not passed to the command when selected.
• resultType (CompletionResultType) - The type of completion result.
• toolTip (String) - The text for the tooltip with details to be displayed about the object. This is visible when the user selects an item after pressing Ctrl+Space.

So it seems like Powershell can fit within rust-driven completions and provide the full feature set.

[epage]

I'm starting small and prototyping for just bash support. Looking at argcomplete, it seems they had to use their own shlex implementation. Hoping we can avoid that. The first step is comex/rust-shlex#12.

[happenslol]

Hi, I'm interested in helping with this this I'm developing a few personal tools using clap that would hugely benefit from dynamic completions. Is there any good issues that nobody is working on you could point me towards?

[epage]

@happenslol Thanks!

Anything unchecked in the "Remaining work" section is up for grabs; just post here that you are getting started on it. The highest priority is the support for each shell as that will help gauge the feasibility and provide feedback on the API. The rest is flushing out parsing implementation.

I should split those out into individual issues to make it easier for people to coordinate on those (and to add bounties) but I probably won't have time for another week or so.

[happenslol]

Alright, had a busy week, so I'm only getting back around to this now.

I've looked at the work done in #3656, and it looks like the next steps to add zsh and fish support would be to pull some of the functionality out of clap_complete::dynamic::bash, and adapt it to the other shells to provide the same functionality. Am I correct in assuming that?

I'd look at argcomplete for the other implementations here, since the current one also seems to be coming from there.

[epage]

@happenslol Yes, that is correct. I'd like to focus on shell support initially as that is where the most unknowns exist

[happenslol]

Started work here. I'm looking at cobra for zsh completions since argcomplete basically takes the easy way out and tells people to enable bash completion support for zsh 😅

[epage]

Oh if cobra is doing the same type of dynamic completions as argcomplete, that is great! That'll serve as a much better example!

[calebbourg]

Hello @epage I'd love to get involved with this. Where might be a good place to provide some value here (something on the easier side to start maybe)?

[epage]

The areas we need the most help are

• Ensure Powershell works
• Identify any missing features between the static and dynamic completions and either note them in the shell specific issues or create new issues (if they apply more generally)
• Figure out clap_complete::env::Elvish generates code deprecated in 0.18, removed in 0.21 #5729
• Testing the various shells and identifying incorrect behavior, particularly around quoting

[daniel5151]

Hey all, just wanted to drop a link to my own hand-rolled implementation of Rust-native argument completion for clap. The code is all MIT licensed. I wrote this to support a feature I wanted in $work project, and now that we went open-source, I can actually link to my effort on this issue.

https://github.com/microsoft/openvmm/blob/main/support/clap_dyn_complete/src/lib.rs

Note that I wrote this code a few years back, before the recent uptick in investments in resolving this issue, and may not reflect the current approach clap_complete is taking.

This implementation is not fully-featured by any means, but it does include support for several different shells (zsh, fish, powershell), various clap constructs, and support for dynamic runtime completions + async support (which $work project uses to dynamically suggest fields based on the return values from async RPC calls).

I'm quite interested in getting this sort of functionality working in clap_complete, but I unfortunately don't have time to do so myself. Folks should feel free to crib any/all functionality from this implementation, in case any of it is useful.

[epage]

Some more quick ones

• Looking over https://github.com/clap-rs/clap/issues?q=is%3Aopen+is%3Aissue+label%3AA-completion+label%3AM-breaking-change and doing any unblocking work on the shell side of these so we can explore what we want the APIs to be
• Exploring a lazier way of doing completions, see Lazy loading of dynamic completions #5668 (comment)

[epage]

@daniel5151 looking over that, some differences in approach

• clap_dyn_complete completes using the index into the string rather than an index into an array of arguments. This allows mid-word completions and I assume avoids COMP_WORDBREAKS problems (Support flags with values in native completions #3920) at the cost of needing to re-implement quoting. I wouldn't be surprised if we need to switch to that but wanting to see how viable our current approach is
• clap_dyn_complete accepts completion information through the CLI. This creates more overhead as you now need your completer to run the CLI parser and enough application startup to get there and it is more rigid for adapting to shell-specific needs or features
• clap_dyn_complete offers native async support. That makes it a lot heavier weight but can offer better concurrency for longer operations
• clap_dyn_complete delay creates custom completers. The use case isn't too clear to me (complete would just be called once, why not just do the expensive stuff there?)
• clap_dyn_complete builds on top of the existing parser to get context, relying on ignore_errors. imo a lot of different parts of the parser shouldn't run (like overrides) and instead we should be parse only whats done. I also don't trust ignore_errors enough to unwrap with it :). So far, we don't provide context though there is an issue for it.

[epage]

@daniel5151 any integrating of ideas from clap_dyn_complete will require someone to drive it, creating issues from anything "needed" from it for the clap_complete solution to include, being able to talk to the problems or use cases that lead to the differences.

[daniel5151]

@epage, glad to see you took a look through the code!

A few additional thoughts, in response to your analysis.

• clap_dyn_complete completes using the index into the string rather than an index into an array of arguments. This allows mid-word completions and I assume avoids COMP_WORDBREAKS problems (Support flags with values in native completions #3920) at the cost of needing to re-implement quoting. I wouldn't be surprised if we need to switch to that but wanting to see how viable our current approach is

This was quite important to me, as the key completion I wanted to support was akin to navigating a virtual directory listing (i.e: completing --remote-path foo/bar/baz, and I wanted to be able to support completions after any intermediate /.

But you're right - I certainly recall wrestling with nuances around quoting and shell arguments. Not sure I ever solved it cleanly.

• clap_dyn_complete offers native async support. That makes it a lot heavier weight but can offer better concurrency for longer operations

IIRC, async support was purely a nice-to-have. I'm sure some more judicious use of block_on could've worked just as well (at least, for my use-case).

• clap_dyn_complete delay creates custom completers. The use case isn't too clear to me (complete would just be called once, why not just do the expensive stuff there?)

I believe part of the rationale here was that I wanted to plumb-through some shared context into different completion handlers without needing to use a global static.

• clap_dyn_complete builds on top of the existing parser to get context, relying on ignore_errors. imo a lot of different parts of the parser shouldn't run (like overrides) and instead we should be parse only whats done. I also don't trust ignore_errors enough to unwrap with it :). So far, we don't provide context though there is an issue for it.

Given that I basically wrote this implementation "for fun" at $work, I wasn't super keen getting bogged down in forking / digging through clap internals to get things up and running. I did the best I could with the API surface that was already available.

And of course - the only performance consideration I had in mind when writing clap_dyn_complete was "does it run well enough for the CLI in our project", which I'm sure wouldn't scale well to sorts of massive CLIs that exist out in the wild.

---

But yeah, ultimately, I only wrote clap_dyn_complete because there was a brief lull at $work, and I thought it'd be a fun little QOL feature to hack on. I'm sure many of my design choices wouldn't translate to a robust, "productized" version of dynamic argument completions.

My goal here was to simply call-out some prior art in this space, and have clap_dyn_complete serve as yet another reference point for any folks actively involved in driving this issue forwards from the clap side.

[shannmu]

@daniel5151 Thank you for sharing. Regarding the issue of allowing completion in the middle of words and with COMP_WORDBREAKS, the current clap_complete doesn't handle it very well. I’ll take a look at your approach.

I’ve tried using bash built-in functions like __init_completions in scripts to mask certain characters, and it successfully entered the common complete logic. However, it seems that bash’s post-processing of completions has some special handling. For example, when completing --flag=, it turns the --flag=bar completion into --flag=--flag=bar.

@epage , I have an important deadline on December 2nd, so during this period, I might only push some of the previous PRs. However, I’ve been following the community’s progress on clap_complete and I will accelerate the effort in December.

And thanks for your involvement, @calebbourg . I’ve contributed to the clap_complete project as well. Feel free to reach out to me via email or zulip https://rust-lang.zulipchat.com/#narrow/channel/421156-gsoc/topic/Project.3A.20Move.20cargo.20shell.20completions.20to.20Rust if you have any questions.