[Proposal] Allow embedding other languages in macros

[tamasfe]

A lot of function-like procedural macros might have syntax alien to Rust (html! Yew macro comes to mind). Highlighing based on expansion is not always reliable as a lot of tokens might not end up in the final code.

So instead of trying to handle every case, I believe there should be an official way to extend rust-analyzer with custom language services and syntax highlights, this would allow library authors to offer better DX for their custom macros.

Proposal

A TextMate scope for macros (VSCode)

A custom scope like meta.macro-function.rust would allow syntax highlighting coming from somewhere else, since scope names cannot be dynamic, the scope should span the entire macro invocation (my_macro!(...)), this way rules can be matched based on the macro's name.

Registering external macro LSPs

External LSPs could be registered for macros and rust-analyzer would forward requests to them. In VSCode, this could be done via the extension API based on this stack overflow answer.

This way rust-analyzer can decide not to forward known macros it can handle, and there are no multiple LSPs processing the same document simultaneously, potentially clashing with each other.

Macros would be processed in the following order:

• Known macros (e.g. println)
• Registered LSP if found by the macro path
• Best-effort based on expansion

Unresolved questions

• I only considered VSCode, as I don't use or know the internals of other editors rust-analyzer supports.
• Macros can be imported by a different name. Rust-analyzer could find out its original name/path, but this wouldn't work for syntax highlights.
• Scrap the TextMate highlight support, and allow macro LSPs to provide semantic tokens.
• This feature would further encourage people to write macros with custom alien syntax, but it's already happening anyway.

[dylanede]

For an extension API, it's also worth thinking about how to support macros that allow embedding of snippets of Rust code inside them. Presumably the macro LSP could provide information about what kind of AST fragment it is expecting in a given location.

[1216892614]

Does anyone know how support for JS and CSS is implemented in HTML?

Can we follow such an approach?

[bjorn3]

JS and CSS are implemented by having the HTML parser contain a couple of special rules surrounding when the <script> and <style> tags end. The DOM then contains unparsed JS and CSS which the browser engine later parses as JS cq CSS when actually interpreting the DOM. For the first point these special parser rules don't exist in rust and IMO shouldn't be introduced either as they are very language specific and would have to be baked directly into the rust parser without allowing proc macros to add such rules in any way. For the second point that is already what we are doing with proc macros. We let the proc macro do all the parsing of the embedded language from token trees and don't feed them pre-parsed code. As for how syntax highlighting with mixed HTML and JS/CSS works is because the HTML syntax highlighting definition has hard-coded forwarding to JS and CSS syntax highlighting definitions. This again is not something that would work for anything but a fixed hard-coded set of embedded languages and thus not usable for proc macro defined embedded languages.

[1216892614]

JS and CSS are implemented by having the HTML parser contain a couple of special rules surrounding when the <script> and <style> tags end. The DOM then contains unparsed JS and CSS which the browser engine later parses as JS cq CSS when actually interpreting the DOM. For the first point these special parser rules don't exist in rust and IMO shouldn't be introduced either as they are very language specific and would have to be baked directly into the rust parser without allowing proc macros to add such rules in any way. For the second point that is already what we are doing with proc macros. We let the proc macro do all the parsing of the embedded language from token trees and don't feed them pre-parsed code. As for how syntax highlighting with mixed HTML and JS/CSS works is because the HTML syntax highlighting definition has hard-coded forwarding to JS and CSS syntax highlighting definitions. This again is not something that would work for anything but a fixed hard-coded set of embedded languages and thus not usable for proc macro defined embedded languages.

I understand what you mean.

But is there a way for me to provide linter support for crates like yew? And how to tell the editor to add a syntax hint to a scope.

Can #7402 really do it all?

[chipnertkj]

I have opened a discussion thread in the Rust Internals Forum related to this issue.
If there is something constructive you could add to the discussion (criticism, possible solutions, concerns, use cases), please head through the link below.
https://internals.rust-lang.org/t/discussion-adding-grammar-information-to-procedural-macros-for-proper-custom-syntax-support-in-the-toolchain/21496