switch to a different tool for python syntax highlighting

[DetachHead]

MagicPython doesn't seem to have been updated in 2 years, meaning it doesn't support any of the new syntax in python 3.12:

• Support for PEP 695 (Type parameters / generics) MagicStack/MagicPython#262
• support python 3.12 type alias syntax MagicStack/MagicPython#265

it seems very odd that these new features came out 6 months ago and are still not supported properly in vscode.

---

there's also the issue with raw strings being incorrectly treated as regex which has been raised many times:

• Wrong highlight for raw string MagicStack/MagicPython#259
• Incorrect syntax coloring for formatted-raw string literals (example: fr"{some_path_here}\some_file_here.txt") MagicStack/MagicPython#149
• Python highlighting - raw string literals incorrectly colour escape sequences in some cases MagicStack/MagicPython#155
• Raw string highlighting MagicStack/MagicPython#114
• Github highlights escape sequences in raw strings.  MagicStack/MagicPython#82

i think this is a big source of confusion for users (as you can see from how many times it's been raised), and should be addressed. i think @gilbertohasnofb makes a good argument in MagicStack/MagicPython#259 (comment):

AFAIK this is not a standard defined in any PEP, and popular code formatter such as black will convert any R'' strings into r'' by default.

I get the reasoning behind this decision, and there were some arguments for it since TextMate and Sublime Text both highlight Python syntax this way by default, but with the widespread use of VSCode these days (which does not highlight raw strings in Python this way by default) this argument seems outdated. Ideally this idiosyncratic highlighting could be made optional, preferably off by default.

[rchiodo]

Thanks for the issue. We actually have an internal issue already tracking this: https://github.com/microsoft/pyrx/issues/4026, but we can use this issue as the external facing problem.

[JelleZijlstra]

AFAIK this is not a standard defined in any PEP, and popular code formatter such as black will convert any R'' strings into r'' by default.

Just noting that this part isn't true; Black does not convert between R'' and r'', specifically because MagicPython syntax highlights them differently. I don't have a strong opinion on whether Pylance should continue to treat the two prefixes differently: on the one hand it's a surprising and non-obvious behavior, on the other hand it is nice to get a way to syntax-highlight regexes.

[DetachHead]

i see this issue was moved to the pylance repo. from my understanding the syntax highlighting provided by magicpython is bundled with vscode and separate to the semantic highlighting provided by pylance. is the plan to remove that basic syntax highlighting entirely and instead have it handled entirely by pylance? if so, it would be a downgrade for users who are not using pylance (eg. vscodium/basedpyright users)

[rchiodo]

The plan is to fix the colors of text in Python code. We haven't really started investigating how or where we'd implement such a thing, but we realize the textmate solution that uses MagicPython's tmlanguage isn't really working out.

[debonte]

VS Code is exploring using Tree-sitter for syntax highlighting. If they move forward with that, the current thinking is that each language could choose either TextMate or Tree-sitter depending on which implementation is more up-to-date / better maintained -- microsoft/vscode#210475

For internal people, there was a discussion about this work at the beginning of the 4/30 "Language Teams and LSP Weekly Sync" meeting.

[heejaechang]

did we drop the idea of supplying textmate rule in pylance rather than using one supplied by vscode?

[debonte]

did we drop the idea of supplying textmate rule in pylance rather than using one supplied by vscode?

No, I think that's still on the table. Although Dirk mentioned in the LSP sync that fixing bugs / enhancing existing TextMate grammars can be incredibly painful.

[JackismyShephard]

What is the ETA on getting these issues fixed? The no syntax highlighting on type definitions and generics is really a thorn in my side 😋

[debonte]

I checked in with vscode on microsoft/vscode#210475. There is no extension contribution story yet.

Our hope is that using tree-sitter will greatly simplify our syntax highlighting approach (vs TextMate), and therefore we are postponing work on syntax highlighting until vscode's tree-sitter experiment is further along.