feature: eglot-execute to optionally redirect executeCommand progress messages to a compilation-mode buffer

[adonovan]

Background: Go's LSP server, gopls, replaces a number of tools that were formerly command-line tools such as gorename. An LSP server has many advantages: it has a hot in-memory cache, it can operate correctly on unsaved editor buffers, and it is of course tightly integrated into the editor. Rename has special support in the LSP. But there are a large variety of analysis tools we would like to build as features of gopls that need only the loosest of integration: at a minimum, the ability to stream text output to a terminal-like viewer, so that file:line:column line prefixes are marked up as source links, as in compilation-mode.

For example, Gopls today has a command, gopls.run_test, that can be invoked via LSP executeCommand. It runs go test on a particular test identified by source line, and sends back the mixed stdout/stderr stream via the LSP progress notification mechanism. Unfortunately, the command is not very useful to eglot users, neither out of the box nor after some elisp hacking, because of the way that eglot currently handles progress messages: it updates the echo area (or optionally the *Messages* buffer) after each notification. To be clear, this is not a criticism of eglot: this is a perfectly reasonable approach. But I think it would enable new categories of tool integration if there were a way to indicate that the output of certain commands should be handled differently: by streaming the output into a new, visible read-only buffer in compilation-mode, much the same way as the compile command streams the output of a subprocess. Initially I imagine that we would have to tell eglot which LSP commands to handle this way, but ideally I would like to make this a core feature of LSP, for example by adding a boolean to executeCommandParams that indicates that the output should be displayed in a buffer. Other editors all have their own UI elements for streaming buffers; for example, VS Code has the Output panel with a selector that chooses among various streams.

Proposal: I propose that eglot-execute be changed so that certain commands are executed with a different progress notification handler that directs their output to a compilation-mode buffer. The set of commands would initially be configured by a customizable variable (i.e. client-side), but perhaps eventually it would be inferred from future metadata in the Command or CodeAction action argument (i.e. under the control of the server).

[joaotavora]

but ideally I would like to make this a core feature of LSP, for example by adding a boolean to executeCommandParams that indicates that the output should be displayed in a buffer.

Yes exactly, though "buffer" wouldn't be exactly the word you'd want to use. Something dumber, more microsoftish or vscodish. In fact, you shouldn't even mention Emacs if you want to increase your chances, which I don't think are very high to begin with (I hope I'm wrong!)

The set of commands would initially be configured by a customizable variable (i.e. client-side),

No, let's use generic functions. Can you show a transcript that very clearly (to save me time) of a LSP/JSONRPC session (real of fictional -- preferably real) that 1. shows an example of such command that wants to "output" to a buffer and 2. another command that doesn't want to output to a buffer. The best shape for the generic function can be decided then.

Or maybe eglot-handle-notification is already the generic function that you want. Keep in mind LSP/JSONRPC is a stateless protocol though.

[nemethf]

It does seem more promising.

He wrote without even checking it out. Seems that LSP upstream shut the idea down :-/ and I also don't see how it could solve the logging outlet problem. But they probably had something figured for that, I guess, right?

go test is probably similar to cargo test, so I think gopls could use experimental/runnables here. Just look at the screenshots of the eglot-x implementation here by searching for "runnables".

Note: it is the client's responsibility to execute the command sent by the server. There is also a test explorer extension where the tests are executed by the server.

[joaotavora]

Note: it is the client's responsibility to execute the command sent by the server.

Ah, then I think it's totally different from what @adonovan is proposing. He wants the server to run the command and send output back to the client. In Alan's model, as far as I can tell, it's the client is merely tasked with displaying the output suitably. This probably makes sense when the server can be "nearer" to the code (At least in Emacs/TRAMP scenarios it it). This model seems interesting, the Rust model -- with all due respect :-) -- seems more useless. Why would I ask the server interface to execute a command that I can ask my own local cargo tool to execute, just for the server to tell me exactly which cargo invocation I should run? Seems needlessly round-trippy.

Anyway, that is besides the point. The bottom line is that this functionality needs a new out-of-protocol notification like $gopls/stdout that is probably not window/logMessage (but could be, if you aded a new out-of-protocol field to that message).

The idea of tokens and creation/deletions of states used in LSP's $/progress can be reused. The only thing missing is a nice indirection between lists of buffers and the tokens/handles used by the new $gopls/stdout messages. That indirection could be, a weak hash table of sorts. Of maybe some slot in a custom CLOS class for gopls. The generic function mechanism of the eglot-handle-notification API should manage that. So it seems this can all be done in some kind of eglot-gopls-extra-stuff.el. Or eglot-x.el, more likely. Not in eglot.el, because eglot.el is, by design, for LSP (in protocol) things.

[adonovan]

I agree that logMessage is not helpful; the $/progress notification mechanism, with its tokens, is what I was referring to. (Strictly, I should have omitted the "Info: all tests passed" line from the reconstituted log since that information comes from logMessage, not $/progress.)

The Rust situation is different, I think: it aims to return commands to the client which would then call execve. In my case, the server is expected to run the command. The command need not be a subprocess (like go test), it could be any server-side function that writes a stream of text followed by a success/failure status.

I don't think the notification mechanism you called $gopls/stdout need be different from $/progress, since progress is already capable of carrying chunks of arbitrary text, and it seems more complicated to define a parallel mechanism. But if it pleases our LSP overlords, it's fine by me.

The only thing missing is a nice indirection between lists of buffers and the tokens/handles used by the new $gopls/stdout messages.

Couldn't it just be a simple global map? If executeCommand knows to wait for the 'start' progress notification for some commands, then it has all the information it needs to deal with the buffer. Cleanup (and map removal) would occur either when the corresponding 'end' notification arrives. Is your concern that the server may forget to send an 'end' message?

[joaotavora]

Couldn't it just be a simple global map?
Is your concern that the server may forget to send an 'end' message?

It could, of course. Do you guarantee cleanup? Weak-keyed hash table? With what key? I think the server slot is easier, but the map works too.

[joaotavora]

I don't think the notification mechanism you called $gopls/stdout need be different from $/progress, since progress is already capable of carrying chunks of arbitrary text, and it seems more complicated to define a parallel mechanism.

Sounds fishy to use $/progress for this... It's main and only use I know is for "Work Done Progress" But you're right there's a separate slot for an "anything", problem is there'snot a very good way to dispatch on that.

https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#progress

There should at least be a kind field of type String in the standard for me to even consider putting something in eglot.el. Until then I recommend using a custom notification and coding a eglot-gopls.el (or adding this kind yourself and then adding an :around just for your server to the method that handles $/progress, but that's slightly harder).