Explore using tree sitter for syntax highlighting

[alexr00]

Goals of the explorations:

• Get Aiday's exploration up and running again: [Exploration] Tree-sitter tokenization exploration (Fixes #161256) #161479
• Come up with ideas for a contribution model
• Confirm performance findings from [Exploration] Tree-sitter tokenization exploration (Fixes #161256) #161479 and understand incremental highlighting

[alexr00]

Some notes on incremental parsing:

• Per Can web-tree-sitter actually do async parsing? tree-sitter/tree-sitter#2118, it looks like we can set a timeout so that each call to parse times-out, with the parsing progress preserved for the next call to parse.
• I've confirmed that this is the case; however, it appears to be broken in the latest version of web-tree-sitter. Going back to 0.22.2 works (latest is 0.22.5).
• There does not appear to be any significant time penalty for setting a timeout.

@hediet FYI.

[alexr00]

Opened tree-sitter/tree-sitter#3341 for the above issue.

[alexr00]

After some discussion, we've decided to first understand the performance of using tree-sitter to parse files. #213565 contains that work.

[alexr00]

We also need new internal editor API to write a more optimal parse function, which @rebornix may help with.

[alexr00]

Progress in July:

• Created a build pipeline for the relevant wasm files (tree-sitter and typescript right now): https://github.com/microsoft/vscode-tree-sitter-wasm
• Published first npm package with said wasm files: https://www.npmjs.com/package/@vscode/tree-sitter-wasm
• Incorporated that package in Add telemtry for how long it takes to parse files with tree-sitter #213565, which is almost ready to merge, but we'll hold off until after endgame as there's no need to rush it in before release.

[alexr00]

Syntax highlighting for typescript using Tree-Sitter can be enabled with:

  "editor.experimental.preferTreeSitter": [
    "typescript"
  ],

There are still bugs, and many themes will not work right.

Next steps:

• Enable colorization tests just like we have for textmate
• Test top themes
• Give the tokenization feature a way to signal that tokens have changed
• Enable nested grammars (ex. regex highlighting in typescript)

[RedCMD]

I don't believe this is correct

vscode/src/vs/editor/browser/services/treeSitter/treeSitterParserService.ts

Line 155 in 11ad426

const newEndPosition = model.getPositionAt(newEndOffset);

model.getPositionAt() is relative to the final document state
when it should be relative to the document during each separate change state

I was having problems in my implementation when having many edits across many lines all at once
I ended up needing to make my own wack .getPositionAt() code
RedCMD/TmLanguage-Syntax-Highlighter@8681d5e#diff-9914030256b6efbf5184ce635cb026f18f21fc51ed548740a07e4972b34ee1a8R380-R389
haven't had an error since

[savetheclocktower]

As a fellow Tree-sitter implementer, I'm duty-bound to point out the Tree-sitter implementation I did for Pulsar (a fork of Atom). Maybe it'll be useful as a second opinion when it's not clear how Zed is doing things.

I had Atom's original Tree-sitter implementation from 2017ish to refer to when writing mine, so it's only fair.

[alexr00]

Updates (though I didn't get to work on this October):

• We have syntax highlighting tests for tree sitter
• Typescript is still the only supported language
• Lots of small typescript improvements

For November, the main plan is performance.