Add more options to playground, directives, format, etc

[slorber]

This PR adds new useful options to have in the playground:

• format md: to easily test the output in CommonMark mode
• directive: to parse syntax such as :::warning
• rehype-raw: useful to reparse raw html string nodes to proper hast tree
• position: to easily strip noise from output AST

Note: I think it's better to not have positions in the AST by default, this creates useless noise and I doubt people use positions in the playground anyway.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
mdx	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 9, 2023 4:14pm

[codecov-commenter]

Codecov Report

Patch and project coverage have no change.

Comparison is base (9eb747d) 100.00% compared to head (07a1ddb) 100.00%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@            Coverage Diff            @@
##              main     #2295   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           22        22           
  Lines         2177      2177           
=========================================
  Hits          2177      2177           

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[slorber]

BTW @wooorm adding this md option is useful for me to better understand how the md parser works.

Currently I'm facing a little Docusaurus issue where html tags are removed (or sanitized) from the output.

For example:

## HTML

### Styling

<span style="color: blue;">blue span</span>

<p style="color: green;">green p</p>

<button style="color: red;">red button</button>

<div style="border: solid; background-color: lime; padding: 10px">
  lime <span style="color: red; margin: 10px;">red</span>
</div>

### Embeds

### Images

#### Closed image tag:

<img src="/img/docusaurus.png"/>

#### Unclosed image tag:

<img src="/img/docusaurus.png">

### Iframe

<iframe src="/"/>

### Security

```md
<p>
  When pressing this button, no alert should be printed
  <button onClick="alert('unsafe');">Click me</button>
</p>
```

<p>
  When pressing this button, no alert should be printed
  <button onClick="alert('unsafe');">Click me</button>
</p>

https://deploy-preview-8960--docusaurus-2.netlify.app/tests/pages/markdown-tests-md

---

I tried to use some options like

        remarkRehypeOptions: {
          allowDangerousHtml: true,
          passThrough: ['div', 'button', 'img', 'iframe', 'html'],
        },

But it doesn't seem to work, see facebook/docusaurus#8960

Any idea how to allow arbitrary html tags? So far it seems only <span> is in the final output, but without the style attribute.

---

Also wondering about the default behavior: considering MDX is a language and already unsafe, doesn't it make sense to allow arbitrary tags in a similar unsafe way when compiling .md files through the MDX package?

I mean, it makes sense to me that we get the same safety level for .md and .mdx files considering we pass the exact same plugin options. It's also easier to explain to users that by default it's not safe in both cases. Eventually, provide opt-in for the safety of .md files.

My feeling is that filtering by default most html tags will lead to many users opening issues on the Docusaurus repo 🤪

[remcohaszing]

Maybe we should add a tooltip here to explain MDX infers this based on the input file extension by default. I think this option makes sense for the playground, but it’s best to discourage using this option.

Actually, maybe a radio button is more appropriate here than a checkbox, although the checkbox currently fits nice in the UI.

[slorber]

yes a radio or select can also work, open to change a bit the UI, not sure how to organize now that it's not just a list of remark plugins.

but it’s best to discourage using this option.

Why? Docusaurus uses MDX and I think supporting a commonmark mode can help existing docs sites to adopt it more easily, ie to have something less strict running asap without a ton of compilation errors to fix

[remcohaszing]

yes a radio or select can also work, open to change a bit the UI, not sure how to organize now that it's not just a list of remark plugins.

Let’s wait for @wooorm’s feedback, as I think he designed the website. I think it looks great already. :)

but it’s best to discourage using this option.

Why? Docusaurus uses MDX and I think supporting a commonmark mode can help existing docs sites to adopt it more easily, ie to have something less strict running asap without a ton of compilation errors to fix

This option changes whether files are treated as commonmark or MDX. Leaving it off, means this is inferred from the file extension. It’s not about strictness (although basically anything is technically valid markdown, but MDX has syntax errors). This option is useful when the file extension is unknown (such as in this playground), but I wouldn’t recommend it otherwise. I don’t know how Docusaurus uses it exactly though.

[slorber]

On my side, I find it helpful to understand the MDX compiler behavior when handling .md files.

I'm not really saying the format option is useful or recommended, and would be fine if instead we had the ability to customize the file name for example, as long as I can easily get the AST for an input parsed with CommonMark compat.

---

Although it's not recommended, Docusaurus historically has many sites that use MDX with a .md extension. One of the first unexpected things people stumble upon while upgrading to MDX 2 is that some tags (images, videos... ) do not appear anymore. See feedback here: facebook/docusaurus#8469 (reply in thread)

For this reason, it's likely I'll add an option format: 'mdx', to allow our users to upgrade their MDX version without having to rename all their files at the same time.

[remcohaszing]

You can pass the filename. compile() takes a VFile compatible. In fact, I recommend always passing a filename if possible. Some plugins may depend on it. For example the JSX dev runtime depends on it.

await compile({
  value: '# Hello <strong>world</strong>!',
  path: '/home/me/hello.mdx'
})

I would also be cautious treating .md files as MDX. MDX treats certain constructs differently.

I.e. given the following content:

<div>
  foo

  bar
</div>

In markdown, this is equivalent to the following HTML:

<div> foo bar </div>

In MDX, this is equivalen to the following JSX:

<div>
  <p>foo</p>
  <p>bar</p>
</div>

I understand it’s useful to add the option for migration purposes in Docusaurus.

---

it’s best to discourage using this option.

I think this may have been misinterpreted. I means it’s best to discourage using this option in general, but there are definitely legitimate use cases. The playground, where there are no actual file names, is a good use case for such an option.

[slorber]

BTW not sure this works for esast, is there a tool to remove esast positions to make exploring the AST more convenient?

I mean these nodes:

{
  start: 68,
  end: 75,
  loc: {
    start: {
      line: 4,
      column: 13,
      offset: 68
    },
    end: {
      line: 4,
      column: 20,
      offset: 75
    }
  }
}

[remcohaszing]

You can try unist-util-position-from-estree.

[wooorm]

@remcohaszing that only gets a position, it doesn’t clear start/end/loc.

@slorber I don’t think there is one, in some tests I’ve used a walker/visit and called delete for those fields

[slorber]

Replaced with:

import {visit as visitEstree} from 'estree-util-visit'

export function removePositionEsast(tree) {
  visitEstree(tree, remove)
  return tree

  function remove(node) {
    delete node.loc
    delete node.start
    delete node.end
    delete node.range
  }
}

[wooorm]

I tried to use some options like […]

passThrough is for node types. There is no button node. That value is a tag name of an element node instead. There are raw nodes. Or mdxJsEsm nodes.

Any idea how to allow arbitrary html tags? So far it seems only <span> is in the final output, but without the style attribute.

If span is there, then your question is answered: they are allowed. No options needed.
However, I do not see span. I don’t think there are any tags.

Also wondering about the default behavior: considering MDX is a language and already unsafe, doesn't it make sense to allow arbitrary tags in a similar unsafe way when compiling .md files through the MDX package?

Tags are allowed. In MDX. But then they are JSX. You are not using the MDX language. You are using the markdown language. Which supports raw strings of HTML inside it.

No framework (react/preact/vue/whatever) understands strings of HTML, they don’t know what to do with them, to illustrate with a common example: "<details><summary>", "</summary>", and "</details>".

The point of MDX is to actually parses JSX and markdown combined.
If you turn off MDX, and still want those strings, you need an HTML parser.
And you could just as well not use MDX. So we don’t by default.

If you want to support HTML in plain markdown, are happy to include more code, and to do more work, then you can use rehype-raw:

mdx/packages/mdx/test/compile.js

Lines 733 to 746 in e77be3a

	assert.equal(
	renderToStaticMarkup(
	React.createElement(
	await run(
	await compile(
	{value: '<q>r</q>', path: 's.md'},
	{rehypePlugins: [rehypeRaw]}
	)
	)
	)
	),
	'<p><q>r</q></p>',
	'should support HTML in markdown w/ `rehype-raw`'
	)

.

[wooorm]

I think a radio toggle like this is better: https://wooorm.com/markdown-tm-language/, https://github.com/wooorm/markdown-tm-language/blob/656e7dea3f5248c728a6eeda4e9780778bd6449f/example/index.jsx#L412-L437. Don’t “connect” it to an option. But allow people to choose whether the input is plain markdown or MDX

[slorber]

Don’t “connect” it to an option. But allow people to choose whether the input is plain markdown or MDX

I'm not sure to understand what you mean here 😅

Updated the PR to this UI

[slorber]

Thanks for the pointers @wooorm , will study all that and see what I'm able to do to support html nodes in Docusaurus .md files ;)

Handled some of the review comments

[slorber]

Thanks @wooorm, was able to make it work in Docusaurus with rehype-raw + passThrough of MDX nodes (some remark plugins like admonition are producing those)

I added remark-raw as a playground option too because I think it can help understand better how it all works:

---

Note: the latest version of rehype-raw@6 still uses hast-util-raw@7 while latest version is hast-util-raw@8 so I didn't get your enhanced MDX error message (syntax-tree/hast-util-raw@8e7f703)

[wooorm]

so I didn't get your enhanced MDX error message

Yep, soon! Always takes time to go through the entire ecosystem ;)

[wooorm]

Some small nits, then ready to go! Appreciate your continued work!

[wooorm]

• Can you remove the export? Don’t think that’s needed
• Can you stick with the style of using calls first, definitions later (somewhere at the bottom probably!)

[slorber]

removed the export, typo

Not sure what you mean by "calls first", isn't it already the case? I used the code style found in unist-util-remove-position

function removePositionEsast(tree) {
  visitEstree(tree, remove)
  return tree

  function remove(node) {
    delete node.loc
    delete node.start
    delete node.end
    delete node.range
  }
}

Or is it the top-level fn declaration that I should put at the bottom of the file?

[wooorm]

Yep, I mean, place it in the highest scope, at the bottom!

[slorber]

Done 👍

But It was already at the highest scope, and useMdx() is at the top. So we have one helper fn above and under the default exported component 🤪

[slorber]

Updated the PR and fixed the merge conflicts

Handled the review, apart the "calls first, defs later" the rest looks good 👍

Let me know if I should change anything else

[wooorm]

Sweeet, thanks for your continued work!
Commented to clarify that Q!

[wooorm]

Awesome, thank you!!