Allow restricting search to crates with binaries for CLI completion purposes

[mqudsi]

Problem

I'm a fish-shell developer, and we have "smarter" cargo completions in addition to those that were previously generated by clap that generate dynamic completions based off the current environment instead of just what's statically baked into the binary via clap (e.g. autocompleting the names of available examples, tests, etc).

I recently added support for autocompleting cargo install ... and cargo add ... by getting crates matching the entered prefix via cargo search but I was wondering if it would be trivial for cargo to support a cargo search --binary-only or cargo search --installable or whatever (even as a hidden/undocumented flag that could change or be removed in the future so it's not a binding commitment) that would restrict the search results to those that contain a binary.

Proposed Solution

I'm not sure if this is technically possible - I would only ask for this feature if such information is readily available from the crate index without introspecting the project itself or doing anything similarly onerous/heavy. Currently we just list all crates and complete both cargo install and cargo add the same way, but with this information we could further restrict completions for cargo install to those that are actually installable.

Notes

If it's not possible it's not a big deal - the only purpose here is to generate more relevant completions since cargo just does a loose string search on the provided keyword and we would be able to generate more applicable results if we could restrict what it returns (e.g. the 10 results cargo search provides are all installable and therefore potential candidates whereas cargo search for certain keywords or partial keywords may return 10 results none of which are installable and therefore none of which are - in the moment - relevant).

[weihanglo]

Thank you for the suggestion. I am a fish user as well ❤️ 🐟

cargo search is in fact a thin wrapper on crates.io API /api/v1/crates. Cargo can't do much on this until crates.io supports the feature you propose. AFAIK the current search functionality in crates.io is built upon PostgreSQL native text search. To extend this, crates.io might need to parse Cargo.toml and store info of installable binaries (bins and examples). I have no idea how hard it would be though.

[epage]

It sounds like you are parsing the output of cargo search but I don't think the output is intended to be parsed and is subject to change. I would be concerned that adding features for programmatic users would endorse that workflow.

Another potential route is the proposed command for the index. The index doesn't have binary information but it would be useful for missing plugin error messages

[mqudsi]

@weihanglo Thanks for being a fish user! The latest fish release has some dynamic completions in place for cargo, but the latest cargo {add, install} completions only just landed in fish master.

With regards to what you shared, IMHO it would be extremely unfortunate if whatever method of searching for crates with binaries confounds installable top-level binaries with installable tests or examples as the latter aren't the semantics of what we'd want to constrain the search to (in terms of making sure the completions for cargo install are relevant to the majority of the people searching for crates matching the search string, where the bulk are looking for crates with first-class binaries (e.g. ripgrep, hyperfine, etc) and not any crate with something that happens to be an executable.

@epage I'm aware that the cargo search output isn't intended to be a fixed, machine-readable format. Unfortunately it's a common hazard writing completions for commands and the majority of CLI utilities do not expose a separate output format for machine ingestion. We are always updating the completions as the output of commands changes over time, either updating the regex patterns to apply across different versions of the output, abandoning old output formats, or gating parsing on the output of the tool version. We don't in any way mean for our completions to ever hinder upstream changes to the output format or even have developers think twice about changing something that some project might be parsing for its reasons, even if it means more work for our team - that's just the nature of the job 🤷

---

This is a completely separate topic but I figure here is as good of a place as any to bring it up rather than filing a new issue - is there any chance of being able to a) restrict searches to crate names (just like pkg, apt, pip, yum, etc can) and b) restrict searches to prefix matches rather than loose fulltext matches? This would be squarely within the postgres wheelhouse and wouldn't need any changes to what's indexed and what's not.

The issue is that cargo search results are less than illuminating for short string sequences, then that's further exacerbated by the fact that our shell completions require prefix matches for dynamically generated completions to show, meaning that some cargo searches may return results that are completely or almost completely disposed of unused. We already limit the minimum length of search strings to prevent abusive requests for searches with meaningless results, but there's still the issue that most of the results returned by a cargo search query for non-specific/non-unique search patterns for crate names generate non-relevant results unless the limit is significantly upped (which we wouldn't do, for many reasons).

As an example, executing cargo search rs brings up two crate with the substring rs in the name (the exact match for crate rs and one crate with a trailing _rs in its name) then none of the remaining 8 results have rs in the crate name and 6 of the results don't even have rs in the name or the description. While here I'm requesting the option to specifically search only crate names, I feel like (and perhaps this should be in a separate issue) the results should be ranked more relevantly, perhaps with crate name matches first, then description:

The problem is that for a query like this, cargo search returns 10 results (with the default limit) but only 1 of them ends up being shown to the user (the first one) because none of the other results have crates whose names start with search pattern.

[epage]

@mqudsi a more sustainable route might be to help towards clap-rs/clap#3166 followed by clap-rs/clap#1232. Clap's currently completion support isn't powerful enough for cargo but I'm hopeful that resolving those issues will make it so. This will then mean that cargo can provide the dynamic completion support itself along with every single command using clap.