[DRAFT] RFC: Api change for mdx/md file parsing

[armano2]

note: this is early draft

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

Description

Problem
In current architecture there is no way to change parser/implementation of parser

• each of plugins require its own implementation of md/mdx parser
• there is no way to change default behavior of parser
• md files are not properly supported
• there is no way to author plugin that changes how markdown files are parsed across website

Solution

• remove custom parsing code from each plugin and all remark/mdx configurations out of them
• create new package @docosaurus/plugin-mdx that is used as "fallback" md/x file parsing

related issues: #5757 #4029

Has this been requested on Canny?

No response

Motivation

support for simple md files

API design

No response

Have you tried building it?

No response

Self-service

• I'd be willing to contribute this feature to Docusaurus myself.

[Josh-Cena]

Yeah, our Markdown infra is quite messy (also #4625)... We have an mdx-loader to load the MD files, a fallback loader when importing Markdown from outside the plugin, and a linkifier in each plugin because only the plugin knows how the MD file is mapped to URL paths. I think we should have a global protocol so plugins can communicate about the file -> URL mapping. I'm aware of the challenges the ts-eslint website is facing

[slorber]

each of plugins require its own implementation of md/mdx parser

Yes, there's a lot of boilerplate atm for plugin authors atm and we should provide better utils to factorize that

there is no way to change default behavior of parser

Yes, we should decouple more from MDX. For example, if your parser emits some kind of MD AST and you want to be able to render it with a runtime lib, this must be possible.

We should:

• not assume that the content prop is a React component
• allow to plugin a renderer of content
• provide a custom parsing hook with unknown return type.

md files are not properly supported

MDX 2 seems to improve on CommonMark compatibility, if this is what you mean:
https://mdxjs.com/blog/v2/#architectural-improvements

there is no way to author plugin that changes how markdown files are parsed across website

Related to #4625

[slorber]

I think we should have a global protocol so plugins can communicate about the file -> URL mapping.

Yes, we need some kind of reverse URL mapping. Currently, one plugin cannot link to another by using md file paths, which is confusing.

[armano2]

MDX 2 seems to improve on CommonMark compatibility, if this is what you mean: https://mdxjs.com/blog/v2/#architectural-improvements

yes its better but MDX 2 removed support for html comments <!-- -->

[armano2]

i just relalased that this is a duplicate of #3018

[Josh-Cena]

Well, I feel like this is a mix of #4625, #3018, and what I'm going to implement in #5568