Move out ALL builtins

[Conni2461]

Disclaimer: This is something TJ just brought up on Discord.

TJ and I think we should move out all builtins to own repositories. like telescope-files.nvim telescope-git.nvim telescope-lsp.nvim etc, ....

Neither TJ nor I enjoy maintaining a lot of the issues that currently are piling up. I only use 5-6 pickers so I am also the wrong person to maintain most of these issues (because i dont have any strong feeling for them) and from my conversation with TJ i think he feels the same.

We can still have them under lua/telescope/builtin/ and we can still support them with :Telescope ... and configuration with telescope.setup{ pickers = { ... } } but with moving them we could gain more flexibility. People can pick whatever picker they need and we (core team and other maintainers) can focus more on the "engine" telescope.

We then can assign more people to these individual repositories.

Thoughts are welcome :)

[clason]

It's a completely understandable endeavor, but let me just note for the sake of completeness that this would add friction to users:

1. For one, the "one stop shop" nature of Telescope is a big selling point.
2. And I just know there will be a constant stream of issues where people forgot to install one of the add-ons and wonder why Telescope find_files isn't working...
3. It also feels wrong to have Telescope without even basic FZF equivalents.
4. And are you sure you want to go full CoC-like "plugin ecosystem"?

Unless you want to pull in the external repos as submodules (including fzf-native)? That might be a compromise here.

[Conni2461]

TJ just mentioned it "out of the blue" and i put it here because to collect more thoughts. So thank you very much :) these are good points.

I am torn apart what i wanna do, we have ~50 pickers and they are a lot to maintain (you know that) and most of the issues that were opened lately didn't interested me.

I dont know if moving them out even has a benefit if no one wants to maintain them in the end we have the same issue.

[tjdevries]

Submodules would be an acceptable solution in my mind. I do want to try and continue to provide out-of-the-box support for a bunch of stuff when you download telescope.

In contrast, fzf the engine (or executable) lives primarily apart from fzf.vim.

I will need to think about it a bit more, but sometimes I get demotivated looking at issues about pickers that I've never used 😆 or trying to weigh in and decide on those (and then some aspect of them leaks into the "engine" or "core" part of telescope, which makes things complicated later).

I will think some more as time goes on :)

[clason]

Let me be clear that I totally get where you are coming from and do not oppose this fundamentally; I just wanted to play advocatus diaboli and add points against this (since you seem to already have points in favor yourself ;))

Also, the example of CoC was not chosen lightly; this could also be seen as a model for where this approach works.

In contrast, fzf the engine (or executable) lives primarily apart from fzf.vim.

That is a common misconception -- fzf (the binary) and fzf.vim (the plugin) actually live in the same repository. What fzf.vim (the repository) contains is additional pickers (basically, accessing vim from fzf, in contrast to the "base" plugin from fzf (the repository) that allows accessing fzf from vim). It's exactly that sort of confusion I would like to avoid ;) (Not that I'd suspect you of that sort of naming flimflam...)

[fdschmidt93]

I dont know if moving them out even has a benefit if no one wants to maintain them in the end we have the same issue.

This is what I'm wary of. At the same time, I'm only at this for two months, maintaining somewhat sizable contributions I don't use myself and trying to keep up with issues for which too often the problem isn't truly clear from the OP; and yeah my excitement is wearing off. To add to what @clason said, I'm also wondering as to whether such fragmentation might cause maintenance problems later on.

What (else) can we maybe do? Just thinking out loud.

• Docs. Telescope is insanely extensible, but writing a custom picker is very challenging for the average user. I'm hoping Rework documentation #1024 will go a long way
• FAQ / Troubleshooting sections: some things come up very often (file_ignore_patterns)
• Github discussions: I find it frustrating that users don't help each other out. I've come to realize Gitter is the wrong platform for usage questions ;)
• Communication: telescope is quite mature; not all issues need fixing and hardly any PR needs to be merged right away.
• More intangible: aggressive and accessible abstractions to the degree, that eg lsp_type_definitions is almost so easy, the average user just writes for themselves. Of course, lots of work which somebody would have to do with uncertain benefit

Playing the long game in hopes more people step up. Then, we can hopefully also gradually expand the team to reduce the burden.

[clason]

I don't know if moving them out even has a benefit if no one wants to maintain them in the end we have the same issue.

☝🏻 This. (And more frustration since your work is now spread over multiple repos, with issues constantly opened in the wrong one etc.)

So that strategy is only really useful if you want to do a hard abandon of Telescope (as a plugin for users rather than a generic framework you personally enjoy tinkering with) -- which is totally within your right, mind you.

So I'm with @fdschmidt93 on this one: the problem is that Telescope is a victim of its meteoric success, with user growth far outstripping contributor growth. So it's a community problem, and hence one that needs to be solved on that level, not with software development -- in a nutshell, you need to be aggressively investing in people.

Some thoughts:

1. Be aggressive in pointing people to Discourse, Matrix (Gitter is officially dead), or Reddit (it's sad to think that so many people need the compensation of imaginary bucks to start helping people...) for usage questions (and err on the side of closing).
2. Pick a core set of pickers (file_find, grep_live, lsp, come to mind) and declare all the other ones "user contributed".
3. Accept enhancement requests but make it clear that there will be no easy instant fix coming from any of you, but you'd be happy to hand-hold people through a PR for it (and follow through on that -- it's not less work in the short run, but hopefully some of these people will stick around).
4. Or be swift in closing them as "no, we don't want that for Telescope".
5. Then look these people up when bugs are reported with these features and ping them to fix them. (Carrot: make them codeowners for that picker, give triage rights etc.)
6. Stick to your guns!
7. Grow the team quickly, but prune if people stop being active (don't respond to pings). (Yes, that includes me.)

[joelpalmer]

This is a great discussion. Good points all around.

The "one-stop-shop" is very convenient, especially when it comes to the most popular pickers. I know my eyes got big a couple of weeks ago when nvim-compe was deprecated in favor of a new plugin with its builtin "sources" now in separate repos. That situation and its motivations are not exactly analogous to this. But, the potential to make the barrier to entry too high for aspiring neoVimmers is present in both cases. Not everyone lives to refactor their Neovim configs like I do. This is especially true for newbies. They haven't been hooked yet.

nvim-cmp will likely be an improvement over compe in the long run by making "source" development easier and maybe by allowing users to decide exactly which sources they want. Well, if they know what they want. A larger ecosystem of sources and a few awesome public dotfiles repos should be enough for folks to show up, overcome and adopt. compe is awesome and cmp is usable and promising. Now, compe is popular but Telescope's "meteoric success" has reached near-ubiquity in Neovimland. That means endless feature requests, support requests, etc. as pointed out so eloquently in the other comments. By the looks of my GitHub Notifications, you'd think this was @Conni2461's full-time job along with others.

I think both of the general approaches mentioned so far have merit and could work. Of course, people will not be motivated to work on features that they don't use or care about. It's difficult to find motivation when you are getting paid to work on things you don't care about. I, for one, ❤️ all the pickers equally 😄. I have complete faith that TJ will set the proper course based on community input and deep thought... or a Twitch poll. I'd be more than happy to take on the Git builtins whether they're in their own repo or stay in core. I really do use more pickers than what I imagine the average user does. So, I would be happy to contribute more in whatever areas need it. I just happen to love Git stuff.

I get so much out of 🔭 and really appreciate the hard work of the team. So, apologies for the long narrative comment instead of concise bullets. I am looking forward to seeing where this goes and helping it get there.

[hrsh7th]

When I was developing nvim-compe, the first built-in source that existed was lsp / buffer / path / vsnip.

While receiving feature-requests, the number of built-in sources gradually increased.
Certainly, submodule may avoid the problem of increasing built-in sources.

google translated

[clason]

I forgot to add this yesterday: Telescope already has the concept of extensions, and those are pretty well-used already. While I strongly believe Telescope core should come with a set of basic pickers, I don't have any argument against doing a round of spring cleaning where some pickers are moved to an extension (which should be sizeable and cohesive, though -- not just one extension per picker!), while at the same time possibly moving extensions into core (coughfzf-nativecough).

[Conni2461]

Thanks for all comments here. You are awesome :)

TJ also had a different idea (tami also suggested it to me). We could also move the engine (core part) in a different repo and provide it as submodule here. So nothing changes for the end user.

We then have to live with that we might need to transfer some issue to the new repo but we can more easily iterate on the core stuff and just test out things because we pinned the version.

[l-kershaw]

I agree with others in that splitting off some of the currently builtin pickers into extensions is likely to cause more maintenance problems later on.

Regarding splitting out the "engine", to me it seems difficult to draw a hard line between "engine" and "not engine". It also has the same problem with there likely ending up being issues in the wrong place, which will require maintainers to spend time doing the frustrating task of sorting it out.

To my mind, the best ideas are:

1. Grow the core team
   • This seems like the only way of keeping maintenance going long term, as there will always be team members who (deservedly) need a break/reduced telescope workload
   • The mentored project system seems like a good way of doing this, but we should be more vocal about it, particularly when closing issues that the core team don't want to work (directly) on.
2. Be clear about what we don't want to work on as a core team, and what we don't want to include in telescope at all
   • This is difficult to do while still maintaining a welcoming environment for new contributors, but being welcoming is particularly important for point 1.
3. Choose a selection of "core" pickers, and have the rest be "user contributed"
   • This makes it easier for us to prioritise issues for the pickers that are more commonly used.
   • This will also help deciding whether a new feature should be part of telescope or an extension.
4. Improve labelling system for issues/prs
   • When looking through all of the issues/prs I often struggle to figure out which ones I can help with. Particularly as the number of open issues has increased, this has become increasingly difficult and overwhelming
   • We would need to keep things simple enough for everyone to understand what is going on, as a lot of labels will get overwhelming fast.
   • Most issues/prs can be categorised by which bit of telescope they are relevant to ("engine", "core"/"user contributed" picker) with a little bit of overlap occasionally
   • We should also label things when saying core team won't work on it/we don't want it in telescope/a pr is in progress
   • We should get rid of any unused labels in the list, to avoid confusion
   • Overall, this will make it easier for contributors to find things that they have the knowledge/willingness to help out on
5. Improved documentation and FAQ/troubleshooting
   • This is underway in Rework documentation #1024
   • As we are trying to reduce the amount of info in the readme, we should make it clear that the docs are the place to look first with a big and obvious link to them. Hopefully this reduces the number of issues where people haven't read the documentation 🤣

Most importantly, I think that contributors (even the more prominent ones) should not feel a responsibility to spend more time working on telescope than they want/are able to. This is a widely used community tool, and it takes a community effort to keep it running smoothly 🙂

[clason]

TL;DR: Leverage Github tooling (CODEOWNERS, Teams, a custom bot) to allow team members to stop subscribing to every issue/PR/commit to Telescope without missing the stuff they are actually interested in. I believe this goes a long way towards staving of burnout and frustration, as the constant stream of notifications is enough to sap anyone's will...

[matu3ba]

I get demotivated looking at issues about pickers that I've never used

Why not make a label the issues by 1. picker/previewer/sorter: problem/enhancement and 2. a proper label for filtering?
The other stuff could be labeled core and have an according description, ie core: problem/enhancement description.

The alternative would be to put all picker picker/previewer/sorter in another repo, but I think it is bad not to have unit tests (of simple real world usage) in the same repository.

[AlexSWall]

From my perspective as someone who would like to contribute but doesn't find it too easy to understand the codebase, I do agree/believe that splitting out the builtins from the core would make it easier and more approachable for new contributors to get involved in the higher-level functionality such as new pickers. I've written my own very simple picker and would be more likely to create such a plugin around it and provide community support for it if this were a fundamental, established way of doing things as opposed to often primarily using builtins.

In essence, I agree with l-kershaw's point on this.

4. Improve labelling system for issues/prs
   • ...
   • Overall, this will make it easier for contributors to find things that they have the knowledge/willingness to help out on

Many good points all around though, this is just one point among many.