Support for Language Server Protocol

[svenefftinge]

##The VSCode team has recently published the version 2.0 of their language-server protocol. The goal is to define common API for editors such as VSCode or Che to interact with language services like

• TypeScript language services
• OmniSharp (C#)
• Tern or the recently announced Orion JS services
• Ensime (for Scala and Java)
• and probably more (please add them to this ticket).

We have started using this protocol for communication between Che and Xtext and would like to move the Che part here and collaborate on the development. @evidolob: could you create a branch, against which we can make a PR with the basic projects we have created so far?

Tasks in this Epic includes part 1 of implementation:

• Dependency: Agent deployment and management Machine Agent specification and implementation #1823
• Language Server Protocol: Respect Language Server capabilities on IDE client Language Server Protocol: Respect Language Server capabilities on IDE client #1789
• Shows errors and warnings inside editor.
• Intellisense - auto-completion in a context aware setting
• Goto Definition [languageserver] Goto Definition #1423
• Language Server Protocol: Find Reference [languageserver] Find Reference #1421
• Language Server Protocol: Implement client part for "Go To Symbol" feature Language Server Protocol: Implement client part for "Go To Symbol" feature #1791
• Workspace symbols to list project-wide symbols matching a query string.
• Language Server Protocol: Implement document formatting Language Server Protocol: Implement document formatting #1788
• Language Server Protocol: Hover feature Language Server Protocol: Hover feature #1800
• Language Server Protocol: Signature Help feature Language Server Protocol: Signature Help feature #1801
• Language Server Protocol: Refactor Symbol navigation UI Language Server Protocol: Refactor Symbol navigation UI #1809
• Language Server Protocol: Support incremental updates [languageserver] Support incremental updates #1425

Language Server Agents.

LS agents are injected into workspace configuration:
“agents” : [“org.eclipse.che.ls.json”, “org.eclipse.che.ls.csharp”]
LS agents and LS binaries are located at remote storage. For the time being it is Codenvy Update Server. Agents are available by the following urls:
https://codenvy.com/update/repository/public/download/org.eclipse.che.ls.json
https://codenvy.com/update/repository/public/download/org.eclipse.che.ls.csharp
When workspace is created agents are being applied. Basically it means agents script execution over running instance. After that LS can be started on demand when user opens files associated with installed LS.

• Dependency: Agent deployment and management Machine Agent specification and implementation #1823

Bug:

Code Assistant and Code Validation don't work for C# #2509

Initialize process:

• User opens file with LS support.
• LS Editor provider calls vs-agent to initialize LS.
• LanguageRegistryService on vs-agent calls LanguageServerRegistry to find and start LS.
• LanguageServerRegistry launch LS.
• When new LS is launched and initialized, LS capabilities passed to IDE.
• IDE use LS capabilities to create editor and open file.
• Language Server Protocol: Respect Language Server capabilities on IDE client Language Server Protocol: Respect Language Server capabilities on IDE client #1789

Diagnostics

RI: C#, JSON

• Shows errors and warnings inside editor.

Completion (IntelliSense)

RI: C#, JSON

• Intellisense - auto-completion in a context aware setting

Is a context-aware code completion feature in some programming environments that speeds up the process of coding applications by reducing typos and other common mistakes. Attempts to do this are usually done through auto completion popups when typing, querying parameters of functions, query hints related to syntax errors, etc.

Goto Definition

RI: C#

• Goto Definition [languageserver] Goto Definition #1423

Perform search for symbol definition and allows to navigate.

Find References

RI: C#

• Language Server Protocol: Find Reference [languageserver] Find Reference #1421

Search project-wide references for the symbol denoted by the given text

Document Symbols

RI: JSON

• Language Server Protocol: Implement client part for "Go To Symbol" feature Language Server Protocol: Implement client part for "Go To Symbol" feature #1791

Allow to view and navigate document symbols. In most cases it’s can be described as flat outline

Workspace Symbols

RI: C#

• Workspace symbox to list project-wide symbols matching a query string.
  Workspace symbols is the list project-wide symbols matching the query string. It’s allow to navigate not only for file or class name but for method or field.

Document Formatting

RI: JSON

• Language Server Protocol: Implement document formatting Language Server Protocol: Implement document formatting #1788

Format all document.

Document Range Formatting

RI: C#, JSON

• Language Server Protocol: Implement document formatting Language Server Protocol: Implement document formatting #1788

Format selected range in document.

Document on Type Formatting

RI: C#

• Language Server Protocol: Implement document formatting Language Server Protocol: Implement document formatting #1788

Automatically invoke formatting after special character typed.

Hover

RI: C#, JSON

• Language Server Protocol: Hover feature Language Server Protocol: Hover feature #1800

The hover shows useful information, such as types of symbols, or, in the case of CSS below, the shape of the HTML that would match a certain CSS rule:

Or for java hover may show javadoc:

Signature Help

RI: C#

• Language Server Protocol: Signature Help feature Language Server Protocol: Signature Help feature #1801

Shows signature information at a given cursor position. For strongly typed programming languages it’s may shows method parameters types:

[svenefftinge]

Also we have defined the interfaces of the protocol in Java. And made it available here : https://github.com/TypeFox/ls-api

[evidolob]

I create branch named "language-server" https://github.com/eclipse/che/tree/language-server

[nadako]

Nice, this is exactly what I had in mind when I started developing haxe-languageserver - that not only VSCode can implement and make use of that protocol, but any editor/IDE! Looking forward for it to be implemented in Eclipse, so I could try adding Haxe support for it, using the language server.

[TylerJewell]

@tolusha @evidolob @gazarenkov @vkuznyetsov - please, can we link in any dependency issues related to agent deployment and end user UX that are related to how a user will add / remove language servers within a workspace? without seeing the full end user workflow issues that we need to handle, it will be hard to understand what the real definition of done is here.

[tolusha]

#1823

[kaloyan-raev]

I want to prototype a PHP IntelliSense support in Che. I want to take advantage of the Crane language server, which provides the PHP IntelliSense in VSCode: https://github.com/HvyIndustries/crane

As far as I understand, the support for the language server protocol in Che is currently a work in progress and it is not available in the master branch yet.

Could you tell me how to setup my dev environment, so I can start experimenting?

[TylerJewell]

@evidolob @tolusha @vparfonov - FYI - can we document some setup tasks for the community to test our work as we go along?

[kaloyan-raev]

I was able to build the branch from the root pom.xml after deleting a couple of test files that were causing compilation errors.

Now I am exploring how the C# support is implemented. I have a Java workspace in my local Che. I created a new C# file. If I open it in the editor, I get an "Initializing Language Server for cs" progress message and the IDE freezes.

It looks like the C# language server cannot be started. Any hints how to troubleshoot?

[kaloyan-raev]

Same story with the JSON language server. There the code for launching the server is simpler and I found that the /home/user/che-agents/ls-json/launch.sh script is missing. I guess this is related to #2032 and these launch.sh are not available yet...

[tolusha]

@TylerJewell
What setup tasks do you mean? Everything is in branch, except @evidolob code part.

@kaloyan-raev
I am checking...

[tolusha]

@kaloyan-raev
I've just fixed agents. No need to pull changes.
json ls requires only a few seconds to be initialized, but c# requires several minutes.
c# ls works properly if you have project.json file in your project.

I tested everything on codenvy/debian_jdk8 container

[kaloyan-raev]

@tolusha Thanks for the magic you have done! I created new project and the Json language server now starts (I would say immediately).

How do I check if the language server works correctly? I don't see any meaningful code assist, and it's hard for me to imagine any for a generic json file.

Does the validation come from the language server too?

[tolusha]

@kaloyan-raev
To use code assist just add any json scheme into your file and press Ctrl+Space
For instance:
{
"$schema": "http://json-schema.org/draft-04/schema#"
}

[kaloyan-raev]

@tolusha Thanks! This works.

I noticed a few issue with the code assist. Perhaps, you already know them:

• The list of proposals is not sorted.
• The list of proposal is not filtered. Regardless the input I type, the proposals list always contains all available proposals and always the first one is selected.
• There is some {{}}} garbage inserted when applying the code assist.

There is also a concept issue with the json editor. I've never seen $schema to be declared for popular json files like bower.json and composer.json. Yet users would expect code assist and validation to work for such files. Is there a way for Che to tell the json language server what schema to use, if there is no $schema declared in the file?

[tolusha]

@kaloyan-raev
Thx.
We aware of them.

[gazarenkov]

Splitted epic moving its second part to #2109 including:

Language Server Protocol: ShowMessage Notification feature #1793
Language Server Protocol: ShowMessage Request feature #1794
Language Server Protocol: Telemetry Notification #1796
Language Server Protocol: DidChangeConfiguration Notification feature #1798
Language Server Protocol: DidChangeWatchedFiles Notification #1799
Language Server Protocol: Document Highlights #1802
Language Server Protocol: Rename feature #1806
Language Server Protocol: Push notification to client on new language server #1427
Language Server Protocol: Ensure order of Events #1426

[TylerJewell]

@slemeur - on the "Document Symbols" - can you please take a look at the UX for the highlight of the typed characters. It is currently a weird yellow box and ends up appearing unnecessarily heavy to the end users. Please identify an improvement here that we can apply.

[kaloyan-raev]

After I've done some refactoring on the Crane language server (HvyIndustries/crane#143), I am now able to run it in Che and get code completion inside the PHP editor.

I prepared PR #2282 with the changes. Please, review and merge it to the language-server branch.

The PR also contains attachments with the agent that must be uploaded to the Codenvy Update Server.

[kaloyan-raev]

BTW, there is a new PHP language server started a few days ago: https://github.com/felixfbecker/php-language-server.

It is implemented in pure PHP. My next goal is to try it out with Che.

Is it be possible to have more than one language servers for the same editor at the same time?

For the moment, the two language server available for PHP does overlap in features, but complement each other. So, it would make sense to have them both available.

[TylerJewell]

Very interesting work! I do not believe it will be possible - for now - to have more than one language server activated for a source file of a particular type / extension. I do not see us working to try and provide support for multiple LS at the same time for a single file. I think we would require users to register / unregister one before activating another. This would be the common scenario, I would expect.

Also - we will be publishing docs later this week or next week around how to author agents, which are ways to package up language servers into hot-deployable services within a workspace. This will be a nice packaging. We haven't done our own language servers this way but need to.

[StevenDoesStuffs]

https://github.com/Microsoft/TypeScript/blob/master/src/server/server.ts
https://github.com/Microsoft/TypeScript/tree/master/src/server

Is that of any significance?

And also, xtext will still be a thing right?

[vparfonov]

Move:

• Language Server Protocol: Code Lens Language Server Protocol: Code Lens #1804
• Language Server Protocol: Code Lens Resolve Language Server Protocol: Code Lens Resolve #1805

to the Support for Language Server Protocol - part 2

[TylerJewell]

@vazexqi - you asked about delivery timeframes - this looks like it will be completed for M2 or M3 of the 5.0.0 release train, which is very near term, likely this month.

[felixfbecker]

Could you add https://github.com/felixfbecker/php-language-server to the list?

[StevenDoesStuffs]

@TylerJewell Any updates on the typescript lang server? Or are we out of luck?

https://github.com/Microsoft/TypeScript/blob/master/src/server/server.ts
https://github.com/Microsoft/TypeScript/tree/master/src/server
Is that of any significance?

[TylerJewell]

@StevenTheEVILZ - we will have to wait and see what Microsoft chooses to do with it. We'll package language servers as they become available.

[StevenDoesStuffs]

@TylerJewell I;m probably missing something, but isn't that licensed under Apache 2.0 like the rest of the typescript package? And also, I'm not even sure if that is a language server; I don't see it using anything from languageserver-node.

[TylerJewell]

Our understanding is that Microsoft has a language server for typescript, but has not chosen to release it yet. It's only for vscode right now. This is the list of language servers that have been published. Not all are open source. https://github.com/Microsoft/language-server-protocol/wiki/Protocol-Implementations

[StevenDoesStuffs]

I see. TY

[StevenDoesStuffs]

@TylerJewell Sorry to bother you again, but then what is the thing I sent you?

[TylerJewell]

I don't know - we are not following what is happening in the typescript libraries. Microsoft will let us know if they add more language servers in the open source realm. We'll also publish the ones that we include soon, too. Unless, of course, you wanted to adapt Typescript into a language server for others to use?

[StevenDoesStuffs]

@TylerJewell Yeah, I was thinking about writing a typescript lang server. I just wanted to make sure that there isn't already one ready to be implemented.

[TylerJewell]

I'd get in touch with Microsoft on their forums and ask about their progress. If they are planning something, you would want to know about it before doing the work.

[StevenDoesStuffs]

K. TY

[tolusha]

@felixfbecker
When we have API and UI to inject ls agents into a machine, the php ls will be among them.
Or are you talking about a bit different list?

[felixfbecker]

Just referring to the list in the issue

[StevenDoesStuffs]

@felixfbecker @TylerJewell I think che is already using crane as its php ls:
#2282

[kaloyan-raev]

@StevenTheEVILZ This is true. I submitted #2282 with the Crane server. That day it was more or less the only available PHP language server.

Since then we have also https://github.com/felixfbecker/php-language-server on the scene. It shows faster progress than Crane so far. Also it is written in pure PHP, which has a greater potential to attract the PHP community for contributions and to reuse existing PHP tools.

I believe very soon we should replace Crane with https://github.com/felixfbecker/php-language-server.

[felixfbecker]

I also use the existing https://github.com/nikic/PHP-Parser for parsing instead of rolling my own, imo this is a much more solid foundation. I also care about as-high-as-possible test coverage.

[TylerJewell]

This is a first cut of the documentation that we need to write around language servers. These are my notes from a 90 minute demo / review with the language server team this morning.

A language server has two phases of management within Che: installation and initialization. The installation is about how the language server gets installed into the stack, so that it is present for Che to turn on or off. The initialization process is the launching of the language server so that it is a running process within the workspace, and that the editor can connect to it.

The installation of a language server must happen within the definition of a workspace, or as part of a stack. Language servers are packaged within "agents" which are scripts that install the langauge server and its dependencies into a workspace. The agent installation can happen either as part of a stack (when the workspace is first started), or dynamically when a file of an associated type is first opened in the editor. It's recommended, for performance, to always include the associated agent within a workspace / stack definition.

Shortly, we will be providing a GUI within the dashboard that lets users add / remove agents to a workspace or a stack. This will also be part of the stack editing process. Stacks can choose from various agents that they want included.

There can be different types of agents added into the system other than language servers. For example, we have an SSH and terminal agent, which provide the services you expect the workspace to have.

Currently, all agents must be pre-defined within Che and saved in this location as part of the Che repository. We are thinking about a public registry for agents where they can be added and removed, but this is a future activity.

Each agent is saved as a JSON file. The name of the file without the extension is the ID for the agent within the agents list used by the dashboard or other products. So the CSharp agent is identified as "org.eclipse.che.csharp". Inside of workspaces and stacks, there is a new data object called agents : [] which is an array of agents that are to be included. The array would list the identifiers of each agent to include.

Agents have a name, a set of other agents that they depend upon, properties and a "script", which defines how the agent's package is to be installed into the workspace. This is not the startup of the agent, but just the installation. The script section of the agent is difficult to read, but we also include the same scripts in this scripts directory and our CI systems convert those scripts to embed them inline within your agent during the packaging phase.

The scripts that you must provide with an agent have a large if block where you provide installation logic for each Linux distribution that we support. You can follow our templates for how to build agents of your own.

Once a language server is included in the workspace as an agent, then there needs to be a launcher that controls when the language server is started and stopped. Technically, agents can be start / stopped for any number of reasons. For language servers, those agents will be typically start / stopped when the editor opens a file of a certain extension or when a plugin of a certain type is created. Right now, launchers need to be authored as Che extensions that are packaged with Che itself. We will also look to make these dynamic in the future.

Write a launcher extension
Add the launcher as a module

Once you have added this extension, you can test your Che by just rebuilding the plugin-language server module and also rebuilding the core assembly.

[kaloyan-raev]

@TylerJewell This was very helpful. Thanks for taking the time to write it down!

[kaloyan-raev]

@TylerJewell It would be helpful to have a wiki page that lists all supported Linux distros and versions, and a minimal Che stack for each of them. This will help testing the LS agent installation and initialization.

[mickaelistria]

I reported microsoft/language-server-protocol#76 about setting up a public registry/marketplace of language servers that all clients could use.

@vazexqi about Che and eclipse-language-server, there isn't going to be much room for collaboration on the UI parts as the UI stacks are quite different. However, there is already collaboration on the common parts such as the protocol itself and the LSP4J api. The good way to move forward if you're interested in LSP is to contribute to both ;)