Deprecating Selectrum in favor of Vertico

[raxod502]

Hey @minad, per radian-software/selectrum#598 (comment) I'm investigating what's needed to deprecate Selectrum in favor of Vertico. Mostly, this means making sure Vertico has feature parity with Selectrum. I figured there would probably be a bit of discussion necessary on a variety of topics, so I opened this issue to have a single place for those.

My first question is about faces used by Vertico. Vertico defines four faces of its own (vertico-multiline, vertico-group-title, vertico-group-separator, vertico-current) but it appears that other faces are used as well, at least by default. I presume these faces come from the default Emacs completion system. For example, here is a dark blue face I'm seeing (with zerodark-theme):

Which doesn't match any of Vertico's faces:

Is there a list of the faces that will end up in the minibuffer by default with Vertico, so that users know what they have to customize in order to change the appearance?

[minad]

The dark blue face you are seeing here is probably the completions-common-part face, which seems unmodified in your zerodark theme. Vertico uses the completion style mechanism for filtering. This means the default faces of the completion styles are used. The default completion styles (basic, substring, flex, initials, partial-completion) use the faces completions-common-part and completions-first-difference. The popular Orderless completion style (which is similar to Prescient filtering) defines the faces orderless-match-face-<n>. Besides that, Vertico uses the default completions-annotations face for annotations, which is equivalent to the selectrum-completion-annotation face.

If you want to use a setup which is comparable to your Selectrum setup I recommend the following:

(vertico-mode)
(setq completion-styles '(orderless basic)) ;; orderless filtering
(savehist-mode) ;; remember minibuffer history, useful for sorting by recency (not frecency)

If Prescient gets completion styles support at some point (radian-software/prescient.el#89) you could integrate that with Vertico and use it for sorting by frecency and filtering. For now I suggest you try Orderless since this package behaves similarly to Prescient.

[noctuid]

If Prescient gets completion styles support at some point (radian-software/prescient.el#89) you could integrate that with Vertico and use it for sorting by frecency and filtering. For now I suggest you try Orderless since this package behaves similarly to Prescient.

I think I asked this before, but I don't remember how the completion system works or if this is actually easily possible. Would it be possible for prescient's sorting and filtering to be implemented separately, so that orderless' filtering could be used with prescient's frecency sorting? Orderless' current style of filtering was inspired by prescient (space separating multiple filters that all much match rather than acting as ".*"), but it is a lot more customizable. It would be great to be able to use orderless with frecency sorting (which I do prefer a little over the recency-only sorting that vertico currently supports).

[minad]

Yes, my preferred solution would be to develop a prescient-new-gen package which takes care of sorting by frecency only. As you wrote, Orderless filtering has advantages over Prescient filtering (flexibility, also performance and it is already a completion style). We can hook Prescient sorting into Vertico quite easily by setting the vertico-sort-function or vertico-sort-override-function. Furthermore Prescient would have to monitor candidate selection. (If I recall correctly, Orderless wasn't inspired by Prescient, it was developed independently. It is more likely that Omar was inspired by Helm and Ivy, which introduced this style of completion earlier.)

Personally I think I don't need sorting by frencency, since I am happy with recency-based sorting so far. Sorting by recency is simpler and more predictable and we can reuse the minibuffer history instead of implementing a custom persistence mechanism. But if someone steps forward extracting the frecency-sorting from Prescient I would appreciate it, since then we would get a solution which can fully substitute Selectrum+Prescient. The great thing about a separate prescient-new-gen package would be that it is separate and orthogonal from filtering and as such fits the spirit of small composable packages better. Frecency non-believers like myself could omit it. ;)

I assume that it should also be very easy to hook the existing Prescient into Vertico+Orderless. See also https://github.com/radian-software/selectrum#alternative-2-orderless, where a configuration for Selectrum is described which uses Orderless and Prescient sorting.

[minad]

See radian-software/radian#505 for @raxod502's current attempt at replacing Selectrum with Vertico in his Radian Emacs config.

[minad]

Btw @noctuid, I've considered before to move out Vertico sorting to a separate extension (see https://github.com/minad/vertico/compare/extract-sort). Sorting is truly a separate feature which does not necessarily have to come with the completion UI directly.

[noctuid]

If I recall correctly, Orderless wasn't inspired by Prescient, it was developed independently.

Right, it was developed independently. I was just talking about the space separating multiple components but was misremembering. No idea whether or not that was inspired by prescient (I think no). It was the added support for initialisms that was inspired by prescient (orderless originally only supported basic regexp components).

Btw @noctuid, I've considered before to move out Vertico sorting to a separate extension (see https://github.com/minad/vertico/compare/extract-sort). Sorting is truly a separate feature which does not necessarily have to come with the completion UI directly

That seems reasonable. So sorting cannot be implemented in some generic way at a level below/separate from vertico then?

[minad]

That seems reasonable. So sorting cannot be implemented in some generic way at a level below/separate from vertico then?

The UI calls the sort function but the sort function itself is mostly agnostic to the UI. My point here is only that the UI doesn't necessarily have to come with sort functionality out of the box so it could be provided by either a small separate extension package like vertico-sort or by prescient.

[minad]

See the comment by @Dima-369 about using Prescient for sorting only in Corfu/Vertico: radian-software/prescient.el#112 (comment). This seems to work without much effort. And Orderless can be used for filtering.

[ghosty141]

Great to hear that sorting is worked on! Would this allow for qualitative sorting? For example when filtering (using orderless) "Interface .cpp" -> in the results, sort "interface.cpp" before "interfaceFoo.cpp".

[minad]

@ghosty141 Sure, this is already possible. In Vertico you can configure your own sorting function, see vertico-sort-function and vertico-sort-override-function. It shouldn't be a problem to implement arbitrary ranking algorithms in these sort functions. Maybe you are also interested in @jojojames' fuzzy completion package.

[haselwarter]

(How) can Vertico implement an equivalent of selectrum-read-file-name ?

For the record, I sorely miss prescient-style frecency sorting from Vertico.

[hmelman]

For the record, I sorely miss prescient-style frecency sorting from Vertico.

If you missed it, savehist-mode comes close

(savehist-mode) ;; remember minibuffer history, useful for sorting by recency (not frecency)

[minad]

@haselwarter Prescient (frencency) sorting can be used in principle with Vertico. There is no need for selectrum-read-file-name. You can simply use the Emacs builtin read-file-name.

[haselwarter]

@haselwarter Prescient (frencency) sorting can be used in principle with Vertico.

@minad That's great, how do I do that? It would be worth documenting this in the readme if it can be made to work without too much difficulty.

There is no need for selectrum-read-file-name. You can simply use the Emacs builtin read-file-name.

@raxod502 can probably explain this better, but selectrum-read-file-name with input /et/x1/ offers me as completions the files in /etc/X11/. Is this achievable with stock read-file-name?

[minad]

@haselwarter I have no recipe on how to connect Prescient with Vertico. But it should be possible. I linked above to another issue where someone tried to connect Vertico/Corfu+Prescient. Overall this needs a little bit of work to make sure that it works smoothly.

To enable the special file completion add partial-completion to the completion-styles. This is documented in the Vertico readme. This is not a Selectrum-specicic feature but already available out of the box by Emacs completion.

[raxod502]

@raxod502 can probably explain this better, but selectrum-read-file-name with input /et/x1/ offers me as completions the files in /etc/X11/

I must confess I actually did not know this was possible! (Although I just tested it and can see that it does.) So, unfortunately, I don't have any insight to shed here.

[minad]

@okamsn and @Dima-369 have added instructions to the Vertico/Corfu wiki on how to connect Prescient sorting to Vertico/Corfu. See also radian-software/prescient.el#112 and radian-software/prescient.el#120.

[velppa]

I tried to switch from Selectrum (with prescient) to Vertico and I see a very noticeable slow-down in showing M-x results. Here is the profiler results:

What can be the reason for that? such slowness is not acceptable as I notice that every time running M-x.

Here's my vertico setup:

(use-package vertico
  :config (vertico-mode 1))

(setq completion-styles '(basic substring partial-completion flex))
(setq read-file-name-completion-ignore-case t
      read-buffer-completion-ignore-case t
      completion-ignore-case t)

[minad]

@velppa The profile you showed here is not suspicious. For Selectrum it will look essentially the same. The time is spent almost entirely during candidate computation (all-completions). For me M-x is fast for both UIs. How does the comparable profile look for Selectrum in your setup?

[sawwheet]

@minad Since updating to emacs 29 i am also experiencing severe lag that i do not notice with selectrum.

secletrum:

        726  70% - command-execute
         724  69%  - byte-code
         724  69%   - read-extended-command
         724  69%    - read-extended-command-1
         724  69%     - selectrum-completing-read
         373  35%      - selectrum--read
         204  19%       - selectrum--update
         138  13%        - selectrum--update-input-changed
         108  10%         - selectrum--update-refined-candidates
         107  10%          - selectrum-refine-candidates-using-completions-styles
         107  10%           - completion-all-completions
         106  10%            - apply
         106  10%             - #<subr completion-all-completions>
         106  10%              - completion--nth-completion
         106  10%               - completion--some
         106  10%                - #<compiled 0x73637a264034283>
         103   9%                 - orderless-all-completions
          20   1%                    orderless-highlight-matches
           1   0%                    orderless--prefix+pattern
           2   0%                 + completion-pcm-all-completions
          30   2%         - selectrum--update-dynamic-candidates
          26   2%          + selectrum--normalize-collection
           4   0%            selectrum--preprocess
          65   6%        + selectrum--insert-candidates
           1   0%        + selectrum--update-window-height
          53   5%       + minibuffer-error-function
           3   0%       - redisplay_internal (C function)
           2   0%        + redisplay--pre-redisplay-functions
           1   0%         marginalia--minibuffer-setup
           1   0%       + timer-event-handler
           1   0%       + command-execute
           1   0%       + minibuffer-inactive-mode
           2   0%  + funcall-interactively
         163  15% + timer-event-handler
         146  14% + ...
           2   0% + redisplay_internal (C function)

vertico:

        1377  99% - command-execute
        1372  99%  - byte-code
        1372  99%   - read-extended-command
        1372  99%    - read-extended-command-1
        1372  99%     - completing-read-default
        1372  99%      - apply
        1372  99%       - vertico--advice
        1271  92%        - #<subr completing-read-default>
        1199  87%         - vertico--exhibit
        1176  85%          - vertico--update-candidates
        1129  81%           - vertico--recompute-candidates
        1116  80%            - vertico--all-completions
        1116  80%             - completion-all-completions
        1116  80%              - apply
        1116  80%               - #<subr completion-all-completions>
        1116  80%                - completion--nth-completion
        1116  80%                 - completion--some
        1116  80%                  - #<compiled -0xce41f757f7c5968>
        1116  80%                   - completion-flex-all-completions
        1116  80%                    - apply
        1116  80%                     - #<subr completion-flex-all-completions>
        1098  79%                      - completion-substring--all-completions
        1092  79%                       - completion-pcm--all-completions
        1092  79%                        - #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_54>
        1092  79%                         - complete-with-action
          56   4%                          - all-completions
          26   1%                           - #<compiled 0x5c9fb82b6610819>
          16   1%                              #<compiled 0xc5a0a517153917e>
           2   0%                            + version-to-list
          17   1%                      + completion-pcm--hilit-commonality
           8   0%            - #<compiled -0x400ec3da98ea5dc>
           7   0%             - sort
           4   0%                #<compiled -0xb723abd9d94de16>
           2   0%            + vertico-sort-history-length-alpha
          30   2%             redisplay
          11   0%           + test-completion
          20   1%          + vertico--arrange-candidates
           1   0%            vertico--display-candidates
          27   1%         + minibuffer-error-function
           1   0%           frame-windows-min-size
           5   0%  + funcall-interactively
           1   0% - redisplay_internal (C function)
           1   0%    eval
           0   0% + ...

[bdarcus]

Since updating to emacs 29 i am also experiencing severe lag that i do not notice with selectrum.

It is possible your build of Emacs 29 is the issue?

I'm on Emacs 29 as well, and I find M-x on vertico instantaneous; no issues at all.

GNU Emacs 29.0.50 (build 1, x86_64-redhat-linux-gnu, GTK+ Version 3.24.34, cairo version 1.17.6) of 2022-07-14

[sawwheet]

GNU Emacs 29.0.50 (build 1, x86_64-apple-darwin20.6.0, NS appkit-2022.70 Version 11.6.7 (Build 20G630)) of 2022-08-25

It does appear to not just be veritco. Using icomplete:

         965  99% - command-execute
         962  99%  - byte-code
         962  99%   - read-extended-command
         962  99%    - read-extended-command-1
         901  93%     - completing-read-default
         676  69%      - icomplete-post-command-hook
         676  69%       - icomplete-exhibit
         617  63%        - icomplete-completions
         586  60%         - icomplete--sorted-completions
         586  60%          - completion-all-sorted-completions
         576  59%           - completion-all-completions
         576  59%            - apply
         576  59%             - #<subr completion-all-completions>
         576  59%              - completion--nth-completion
         576  59%               - completion--some
         576  59%                - #<compiled -0xce41f757f7c5968>
         576  59%                 - completion-flex-all-completions
         572  59%                  - completion-substring--all-completions
         569  58%                   - completion-pcm--all-completions
         569  58%                    - #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_54>
         569  58%                     - complete-with-action
          22   2%                      - all-completions
          16   1%                       - #<compiled 0x5c9fb82b6610819>
           8   0%                          #<compiled 0xc5a0a517153917e>
           1   0%                        + version-to-list
           4   0%                  + completion-pcm--hilit-commonality
           2   0%             minibuffer--sort-by-position
           1   0%           + #<compiled -0x400ec3da98ea5dc>
          18   1%         + icomplete--render-vertical
          22   2%          sit-for
         156  16%      + command-execute
          37   3%      + minibuffer-error-function
           2   0%      + redisplay_internal (C function)
           1   0%      + minibuffer-inactive-mode
           1   0%      + minibuffer-mode
           1   0%      + timer-event-handler
           3   0%  - funcall-interactively
           3   0%   + execute-extended-command
           2   0% + redisplay_internal (C function)
           0   0% + ...

I realze this might not be the thread to be posting these things, but maybe could help others if they stumble upon it too.
But company has become laggy on elisp files as well (but fine in lsp):

         637  67% - company-post-command
         612  64%  - company--perform
         589  62%   - company--continue
         587  62%    - company-calculate-candidates
         587  62%     - company--fetch-candidates
         587  62%      - company-call-backend-raw
         587  62%       - apply
         587  62%        - company-capf
         587  62%         - company-capf--candidates
         587  62%          - completion-all-completions
         587  62%           - apply
         587  62%            - #<subr completion-all-completions>
         587  62%             - completion--nth-completion
         587  62%              - completion--some
         587  62%               - #<compiled -0x563074662b4bd7a>
         572  60%                - orderless-all-completions
           4   0%                   orderless-highlight-matches
           1   0%                   orderless-filter
          15   1%                + completion-pcm-all-completions
           2   0%    + company--continue-failed
          23   2%   - company-call-frontends
          23   2%    + company-pseudo-tooltip-unless-just-one-frontend
          25   2%  + company-call-frontends
         179  18% + timer-event-handler
         120  12% + command-execute
           5   0% + redisplay_internal (C function)
           3   0% + evil-escape-pre-command-hook
           1   0%   winner-save-old-configurations
           0   0% + ...

Removing orderless has seemed to fix this

[minad]

@sawwheet Please report this upstream or ask on the emacs mailing list about this. This is a regression in Emacs 29 if multiple packages are affected by this inefficiency. The issue might be that Emacs 29 added an expensive version check in the completion predicate, see https://github.com/emacs-mirror/emacs/blob/4015d561c39d29200bf1ac542844fd5f3ba99426/lisp/simple.el#L2334. Can you create a minimal reproducible starting from emacs -Q? Only Icomplete/Vertico with or without Orderless? Narrowing down the precise commit in Emacs 29 which caused this via git-bisect could also help.

[minad]

@okamsn Regarding the toggles in selectrum-prescient/vertico-prescient, isn't it possible to implement this in an UI agnostic way? I think UI specific code is only needed to refresh the UI (the call to vertico--exhibit).

[okamsn]

@okamsn

For using the Presicent sorting with Vertico, I would recommend setting vertico-sort-function to prescient-completion-sort, to use Prescient frecency sorting for all tables that don't already specify a sort, if that is the question. Selectrum+Prescient behave similarly, I think.

It should work the same way if you set prescient-completion-enable-sort=t. Doesn't it?

Only for candidates that were filtered by prescient. The fallback/override styles won't be sorted. There's also the matter of company-prescient re-sorting candidates from company-capf.

Because of those things, I think that it makes better sense to control the sorting at the interface level instead of the completion-style level.

In the README, I mention a custom sorting function because I don't know whether all UIs work by applying sorting after filtering and not stripping the text properties from candidates, if that is the question.

Yes, all UIs must apply sorting after filtering. This order is required by the completion table API. (Selectrum deviates in this point which leads to the compatibility issues and bugs mentioned in the wiki.)

That's good to know. I will simplify the docs with that in mind.

Regarding the toggles in selectrum-prescient/vertico-prescient, isn't it possible to implement this in an UI agnostic way? I think UI specific code is only needed to refresh the UI (the call to vertico--exhibit).

I agree, so long as the variables can be made local to the UI's buffer. I will check whether advising such generic commands to run the UI-specific code is enough. The modes could just add/remove the advice.

[minad]

@okamsn

Only for candidates that were filtered by prescient. The fallback/override styles won't be sorted. There's also the matter of company-prescient re-sorting candidates from company-capf.

Because of those things, I think that it makes better sense to control the sorting at the interface level instead of the completion-style level.

Alright, I missed that you want to use override styles. Then I suggest to remove prescient-enable-completion-sort and the metadata adjustment entirely from Prescient. It is better to offer a single technique which works everywhere. This keeps Prescient simpler, given that the metadata adjustment is messy. Furthermore we follow the design of Orderless/Vertico more closely, since sorting and filtering are decoupled in this case. See also #237 (comment).

[raxod502]

Ok, I've merged my PR updating Radian to use Vertico instead of Selectrum, and have been using Vertico personally for the last few weeks. Is the next step to update the Selectrum README noting that Selectrum is soft-deprecated, and linking to the migration guide?

[minad]

@raxod502 Yes, this is the next step. I am sure we will receive more feedback then, such that we can extend the migration guide.

[raxod502]

Ok cool. Although I guess it makes sense to wait until radian-software/prescient.el#131 is merged, so we can point people at that.

[minad]

To give you a status update:

@okamsn added the vertico-prescient-mode and corfu-prescient-mode to the prescient repository and updated the wikis including the migration guide accordingly. The integration packages vertico-prescient and corfu-prescient are already available on MELPA.

[hmelman]

I'm not sure if it is possible to use both orderless for filtering and prescient for sorting, but either way I think it's a question that people will have that should be documented some place and I don't see it now.

[minad]

@hmelman Yes, it is possible to enable only prescient sorting. See the documentation https://github.com/radian-software/prescient.el#for-vertico.

[hmelman]

@minad Good to know and see. It would be nice if orderless was specifically mentioned so someone search for "using prescient with orderless" would be directed there.

[minad]

@hmelman Sure. In case anyone is interested in such a configuration and has it running in their setup, I'd appreciate a contribution to the wiki. The wikis are open for everyone. But I suspect that Orderless+Prescient is a fringe configuration which isn't very popular given that Vertico already sorts the candidates by recency out of the box.

[minad]

@raxod502 Did you try updating your radian configuration to use vertico-prescient-mode? Shall we proceed with adding a note to the Selectrum README as you proposed in #237 (comment)?

[okamsn]

@raxod502, As a heads-up, I (and maybe someone else in radian-software/prescient.el#137) had install issues arising from the different URL's in the old version of Prescient's recipe and Vertico/Corfu-Prescient's recipe.

I just made Straight redownload everything, but there's probably a better way.

[raxod502]

Yep, I didn't get to this last week but here you go! radian-software/selectrum#601

Thank you very much for the help! Let me know what you think of the update, feel free to adjust and merge as desired.

I personally don't have recipe issues with the latest MELPA version, but I can very much believe that's because I instinctively knew the right sequence of knobs to twiddle. There are issues in this area for straight.el, for sure.

[raxod502]

I guess that is that, then. At some point when I have more time (I'm focusing on improving maintainability for a different of my projects right now), I will go through Selectrum and help contribute to the migration guide, potentially rewrite Selectrum as a Vertico shim, basically finalize the process of getting everyone off the old version, and go through and clear out the issue tracker.

[minad]

@raxod502 Thank you, looks good. I merged the PR. I've also considered creating a shim. It may suffice to not emulate all Selectrum features 1 by 1. Instead selectrum-mode could simply enable Vertico with reasonable defaults and print a warning message.

[ashton314]

If I may interject for a moment, I'd like to convey my sincere thanks to @raxod502 and @minad for your contributions to Emacs. Selectrum taught me to love lightweight packages, and Vertico just added onto that. So thank you, both of you.