completions: Support adding additional code to complete values

[joshtriplett]

Closing comment:

We currently provide ValueHint for leveraging shell logic, though it isn't customizable. For that, we have #1232.

I'm inclined to close this in favor of wrapping up this request with #1232. I'm hesitant for us to have two different solutions (a Rust driven and shell-specific snippets) and doing shell snippets would couple args to completion so they would know what shell they are completing for.

We can always expand how we are using ValueHint for the common cases and as a short term until we have full customization.

---

For some argument values, the bash-completions may want to include additional logic for what type of value to complete, rather than allowing arbitrary strings. For instance, an option might accept a path to a file that exists; bash has a mechanism for that. Or, an option might accept the name of a git ref that exists; that's something more easily implemented in the program in Rust. Either way, it makes sense to augment clap's completion logic.

This also argues for implementing the completions by calling the program at runtime, rather than via shell; that way, arbitrary logic in Rust can determine what to complete, rather than providing a template system to feed in shell code (ick).

[kbknapp]

Perhaps adding something like was discussed in in #376 where there is a "completer" function? I'm all for this, but figured it's also addable in a backwards compatible way once I had the base implementation complete.

I'm just not sure which would be the best way to add this so I'm open to all ideas.

[joshtriplett]

I'm honestly not sure either. I'm really hesitant to suggest inlining shell script snippets into Rust code as strings; I'd rather see those written in a separate shell file and included from there (not least of which to get the right filetype and highlighting). bash (via compgen) and bash-completion (via functions in /usr/share/bash-completion/bash_completion) have some built-in helpers, and it'd be nice to support those for common cases like hostnames, users, and files (with glob patterns). Someone might also want to write arbitrary shell code to enumerate argument values. It'd also be nice to support using arbitrary Rust code by invoking the program.

I think what I'd suggest is that the .completer function should take an enum argument, where that enum has values like User, File, FileGlob("*.ext"), BashFunc("__comp_function_name"), and RustFunc(...). Those would then translate into appropriate calls to compgen, calls to the specified function, or invocations of the program to run Rust code. (That last one would also require something like a global_setting to enable a --clap-complete option or similar.)

This is turning out to be a remarkably hairy yak.

[mathstuf]

In zsh at least, clap could generate completion function calls such as:

(( $+functions[_appname_clap_complete_ARG] )) || _appname_clap_complete_ARG () {
}

Which can then be overridden in a supplemental file included before this one (via source if the file exists). Bash probably has some mechanism that works similarly.

[emk]

I've just converted cage to use clap, and I'm very happy with the results. Basic completion works under both bash and fish. Great code!

But cage would benefit enormously from being able to dynamically complete the names of docker services for commands like:

cage test $SERVICE_NAME

If the project in the current directory has the ervices foo and frontend/bar I would like to be able to do the following:

> cage test f<TAB>
foo
frontend/bar

I would be happy to add an extra argument to the app, something like:

> cage --_complete-service f
foo
frontend/bar

And declare this as:

- SERVICE:
    value_name: "SERVICE"
    required: true
    complete_with: "_complete-service"
    help: "The name of the service in which to run a command"

Obviously the details could vary a bit, but we would ultimately have --_complete-pod, --_complete-service, --_complete-pod-or-service and --_complete-repo-alias, among others. Also note that many different subcommands would share each completion hook, which might mean we want these to be potentially global.

[kbknapp]

@emk Your post has me thinking about this more and more, I'm thinking some sort of hybrid between what @joshtriplett listed above and what you're proposing.

My schedule is pretty busy this week, but I should be able to at least test some ideas and see the feasibility. Stay tuned to this thread for updates!

[emk]

Another approach worth a quick glance might be optcomplete for Python: http://furius.ca/optcomplete/ As far as I can tell, this uses one small, universal shell script for each supported shell, and offloads all the actual completion work to the application's own arg parsing machinery. I think another Python library just uses a '--_complete' on the program that does all the actual work, but I can't find it right now. I'll keep Googling around when I have moment and post anything interesting I find. Thank you so much for a great library and for looking into this!

[emk]

Ah, here we go. Some docs on how several Python arg parsing libraries handle --_completion.

selfcompletion is a layer on top of argparse to take the fine-grained model
argparse builds of the arguments your program accepts and automatically
generate an extra '--_completion' argument that generates all possible
completions given a partial command line as a string.

The '--_completion' argument in turn is used by a generic bash programmable
completion script that tries '--_completion' on any program that doesn't have
its own completion already available, renders the output of the program's
built-in completion if available, and otherwise silently falls back to the
shell default.

Here is the generic completion function for bash:

_foo()
{
    prog="$1"
    while read glob_str; do
        case $prog in
        $glob_str)
            return 1;;
        esac
    done < <( echo "$SELFCOMPLETION_EXCLUSIONS" )
    which "$prog" >/dev/null || return 1
    _COMP_OUTPUTSTR="$( $prog --_completion "${COMP_WORDS[*]}" 2>/dev/null )"
    if test $? -ne 0; then
        return 1
    fi
    readarray -t COMPREPLY < <( echo -n "$_COMP_OUTPUTSTR" )
}

complete -o default -o nospace -D -F _foo

The advantage of this approach is that the per-shell code can be written only once, and all the hard work can be done directly by the application itself. Obviously, there might be disadvantages as well. But I figured it was worth tracking down all the existing attempts to standardize this to see if any of them had helpful ideas. :-)

[joshtriplett]

@kbknapp Any updates on this mechanism? I have someone asking after completions, and I'd love to beta-test this.

[jcreekmore]

@kbknapp I would be interested in this as well. I am currently post-processing my completions to substitute in _filedir for filename completion, but that is less than ideal.

[kbknapp]

@joshtriplett @jcreekmore

Now that the ZSH implementation is complete I've got a better handle on this. The biggest issue I see holding this up is that completions are done differently between all three (so far) supported shells.

I'm all for some sort of enum with variants that allow things like, Files, Directories, Globs(&str), Code(&str) or something to that effect. But some shells support those things verbatim, others only in arbitrary ways that clap doesn't use when gen'ing the completion code.

I'm just unsure of the best way to expose this. Perhaps on an Arg::complete_with(enum)?

I guess, first what is the shell you're trying to support, and what particular portions are you wanting to inject into the completion code?

[emk]

@kbknapp For cage, we'd like to be able to complete custom "types" of values, such as Docker container names, "pod" names, target environments, and so on. The legal values can only be determined by asking our executable at runtime, since they vary from project to project.

This is pretty much how git-completion handles origin names, branch names, etc.

The problem with an enum is that it would limit us to just a few built-in types such as Files, Directories, etc., right?

[joshtriplett]

@kbknapp I don't think you need to support embedding arbitrary shell code from Rust. My suggestion would be to support the lowest-common-denominator bits (filenames, usernames, filenames matching one of these patterns, etc), and then have a "call this Rust function" variant that invokes the program itself with some internal hidden --clap-complete option that dispatches to that Rust function. That makes it easy to do things like "a git ref matching this pattern", by calling a Rust function implementing that.

For those common categories like filenames or usernames, use the shell built-in mechanisms if available, or provide and use code implementing those simple completions if the shell doesn't have them.

If people want "invoke this shell code", I'd suggest adding a Rust variant to call a named shell function, and then letting people add that shell function to the resulting generated completions for any shell they support. That seems preferable to embedding many variants of shell code directly.

enum Completion<F: Fn(...) -> ...> {
    File,
    User,
    Hostname,
    Strings(&[str]),
    Globs(&[str]),
    ShellFunction(&str), // maybe
    RustFunction(F),
}

[kbknapp]

@emk Yes, and no. It would be extensible, so more variants could be added. But Some of the variants could also take additional parameters, and ultimately (possibly) injecting arbitrary shell script via something like ,Code(&str) which of course isn't super great, but perhaps a fallback if a particular variant doesn't quite fit the bill. (Or perhaps not...it could end up being massively unsafe 😜 )

At the same time, I haven't looked into exactly where this code would be injected and ultimately if it's even feasible yet. This is just straight of the top of my head right now.

Also, if the "types" are know prior to runtime ZSH already supports this just by using the Arg::possible_values

[joshtriplett]

That's true, "one of these fixed strings" should be an option as well. Updating the type in my previous comment.

[kbknapp]

From @Xion in #816

In Python, there is a package called argcomplete which provides very flexible autocompletion for apps that use the standard argparse module. What it allows is to implement a custom completion provider: essentially a piece of your own code that's executed when the the binary is invoked in a Special Way (tm) by the shell-specific completion script.
For an example, see here. The code is preparing completions dynamically from the filesystem, or even from a remote API (if certain flag isn't passed (flags are partially parsed at this point)).
Having something like this in clap would be very nice. I know this is a potentially complex subsystem so it'd be unreasonable to expect it implemented anytime, but I wanted to at least put this feature on the radar.

[kbknapp]

After reading through some of the argcomplete python module the hardest part will be figuring out how to call a Rust function from the shell.

[kbknapp]

I'm guessing what'll end up happening is some sort of double run with hidden args.

[kbknapp]

Expanding on the ideas (from #818)

The problem with implementing this is I just haven't had a good time to sit down and think about how (because of work, holidays, family, etc.). I want a way to specify this that abstracts well enough to work for all shells. The easiest way is to say, "Put your arbitrary completion shell script here inside this Arg::complete_with(&str)" but that feels super hacky to me, and potentially unsafe. What I'd like to do is provide a Arg::complete_with(Fn(&str, &str)->String) (and a Arg::complete_with_os(Fn(&str,&OsStr)->OsString) where an arbitrary Rust function is called...but herein lies the problem; shell completions are run before the program executes. This has led to some people using hidden args or something like, $ prog --complete me<tab> calling a shell completer that actually runs $ prog --_complete_arg "complete" --_complete_prefix "me" which generates the possible completions and returns them to the shell. I'm not against doing that, but again feels strange because you're injecting hidden args into a CLI. Although typing this out right now does make me lean towards this solution.

[Dylan-DPC-zz]

@NotBad4U you can start working on it. if you need any help you can ask us in #wg-cli channel on discord.

[zx8]

https://github.com/posener/complete is a library dedicated to dynamic completion written in Go, if you're looking for some ideas/inspiration. Despite the repo's description, it supports bash, zsh & fish.

[benjumanji]

Here is a proposal that I think would be relatively simple. I haven't checked how zsh completion is done, but it's my understanding that it understands bash so you could just re-use it?

The generated scripts for both fish and bash already support dynamic completions. Both compgen for bash and complete for fish honor parameter expansion, including shell substitution and word split. That is to say if you throw $(cat words) for bash or (cat words) for possible_value into an arg it totally works! Right up until clap validation kicks in and says it's not a valid value.

Proposal: add dynamic to arg which adds the shell code and disables value validation. I can't justify working on this right now, but if this is an acceptable proposal I might be able to circle back around to it in a few weeks when I might need it.

[8573]

I haven't checked how zsh completion is done, but it's my understanding that it understands bash so you could just re-use it?

I've forgotten almost all I once knew about the Zsh completion system, but I recall that it's much more powerful/expressive(/fancy) than Bash's such that, if I recall correctly, telling it to use Bash completion, while the easy way out, could provide a needlessly suboptimal user experience. That said, I would think that such a project could start with telling Zsh to use Bash completion and come back and write native Zsh completion later.

[pksunkara]

#1793 is an attempt at fixing this in zsh.

[epage]

Are #1232 and this now dupes? Should we close one in favor of the other to make it easier to browse the backlog?

[pksunkara]

I kept them separate because this issue is more concentrated on leveraging shell completions while the other is for a full fledged solution.

[Milo123459]

I'm looking to try and implement this. Is there an API in specific that should be used?

[epage]

We currently provide ValueHint for leveraging shell logic, though it isn't customizable. For that, we have #1232.

I'm inclined to close this in favor of wrapping up this request with #1232. I'm hesitant for us to have two different solutions (a Rust driven and shell-specific snippets) and doing shell snippets would couple args to completion so they would know what shell they are completing for.

We can always expand how we are using ValueHint for the common cases and as a short term until we have full customization.

If there is any concern with closing this, please let us know!