hast-util-to-mdast - adding custom mdast nodes

[TRIAEIOU]

Hi,

I am trying to extend mdast with some additional node types (corresponding to html nodes such as subscript, superscript etc.) and all is well except when using document fragments where the original html does not wrap all text in paragraphs, e.g. some text <sup>that is superscript</sup> and so on. This example results in superscript being converted to a sibling to the initial paragraph node rather than a child (i.e. [paragraph: {}, superscript: {}, paragraph: {}] rather than [paragraph: {[text, superscript, text]}]).

Stepping through the code it seems the issue lies in mdast-util-phrasing where any custom node fails the phrasing() in wrap.js resulting in wrapNeeded returning true and the custom node ending up getting wrapped separately.

What is the correct way to extend the list of phrasing nodes to include the custom nodes?

export const phrasing = /** @type {AssertPredicatePhrasing} */ (
  convert([
    'break',
    'delete',
    'emphasis',
    'footnote',
    'footnoteReference',
    'image',
    'imageReference',
    'inlineCode',
    'link',
    'linkReference',
    'strong',
    'text'
  ])
)

Or am I barking up the wrong tree?

Thanks!

[wooorm]

Specific to that project, it relates to “wrapping”: https://github.com/syntax-tree/hast-util-to-mdast/blob/main/lib/util/wrap.js. Right?

That list currently, as you mention, uses mdast-util-phrasing: phrasing nodes in mdast. That list currently can’t be extended.

One solution is to add an options.phrasing. But, that’s probably so niche that it is not useful to many folks and if it would be useful to them, they probably wouldn’t find it.

There is a similar list, hast-util-phrasing, which works on HTML tag names. The wrapping doesn’t operate on that, but it might be interesting.

What, if you attach HTML information to your custom nodes? That information is useful for the reverse process: https://github.com/syntax-tree/mdast-util-to-hast#fields-on-nodes. It makes sure your custom nodes integrate with many utilities that typically work with HTML.

We could, when wrapping, look for this information, and compare it to hast-util-phrasing?

[wooorm]

is that on the mdast node?

Yes.

And then the idea would be for mdast-util-to-markdown/toMarkdown would this into account?

No.

Or is the idea to embed info on the hast node and then modify hast-util-to-mdast/wrap.js/wrapNeeded and hast-util-to-mdast/wrap.js/runs to both use mdast-util-phrasing/phrasing() and the hast node properties to determine whether to wrap the node separately?

Yes. It can then use hast-util-phrasing, and might still need mdast-util-phrasing too.

the issue arises when converting hast nodes to mdast nodes, the hast nodes themselves are fine (as they are standard HTML nodes) and the conversion mdast to hast is also fine.

Yep, hast nodes are fine!
Known mdast nodes are fine as well.
But nobody knows how to handle undefined mdast nodes.

As these mdast nodes have a hast representation, we can embed that hast representation on the mdast nodes.
That happens through fields. Sort of like your contentType idea. But it doesn’t embed that. It embeds the HTML tag name. That HTML tag name can be used to infer the content type.
This is how math syntax in markdown works too: nobody knows how to handle math. But because some information on how to handle it is embedded on the node, it can be handled!

This is great because you should probably add this info anyway.
Presumably, you have custom nodes so you can a) parse to them (with micromark extensions), b) convert them (with mdast-util-to-hast), or c) serialize them to a custom markdown construct (by extending mdast-util-to-markdown).

In the case of a), other people will want to do b).

In the case of b), those embedded fields will tell mdast-util-to-hast and all other hast-aware utilities to handle them.

For more background, I recommend the previously linked https://github.com/syntax-tree/mdast-util-to-hast#fields-on-nodes! Some more on why there are two different ecosystems is written here: https://github.com/remarkjs/remark-rehype#what-is-this.

[TRIAEIOU]

Ok, I think I understand, so something like this?

At a conceptual level there is something I am missing in your argument against including some possibility to define where in the mdast content model custom node types belong (whether that be by extending the PhrasingContent/FlowContent/ListContent definitions, define on the respective node or or something else) but that is fine.

[wooorm]

Yup, something like that! Maybe as a reusable function?

function phrasing(node) {
    return node.data && node.data.hName
        ? hastPhrasing({type: 'element', tagName: node.data.hName, properties: {}, children: []})
        : mdastPhrasing(node);
}

---

At a conceptual level there is something I am missing in your argument [...] but that is fine.

I’ll try once more! :)

As concretely as I can:

1. Phrasing is a concept that comes from HTML
2. Your custom nodes relate to concepts from HTML too.
3. Any custom node is ignored everywhere. The way to get them seen by any tool that understand HTML too, is with h* fields.

You should do 3, for your custom HTML nodes, to integrate with all other tools.

So why invent another field? If we already have a solution?

[TRIAEIOU]

Yup, something like that! Maybe as a reusable function?

Ok, I'll implement your suggestion, do you want me to submit a PR once done?

---

As concretely as I can: [...] So why invent another field? If we already have a solution?

And my (irrelevant) conceptual counter argument is that phrasing may come from HTML but it is a part of the mdast model (and different types of elements/contents are not really foreign to markdown - block vs span elements) so to me it seems cleaner to add a custom node and define where in the content model it belongs. Especially as it seems to me it is the only thing missing to make it a "full" part of the mdast model (I am not talking about the serialization of mdast etc).

For instance, the gfmStrikethrough extension depends on del already being hard coded in the PhrasingContent list despite not being a part of core markdown - conceptually it should not need to depend on special treatment.

Now, my suggestions on how to add a custom node to the content model might be stupid (from a conceptual level adding properties to nodes seems not the cleanest way to my newbie eye, being able to extend PhrasingContent/FlowContent/ListContent would be more conceptually correct, I only suggested the property-thing because I guessed it would be easier to implement). I also perfectly understand that realities (harder to maintain, expanded code base, degraded performance, a lot of work and testing for very limited use etc.) may result in it being more practical to use the HTML tag solution.

But, all of the above is not important, it is just a conceptual discussion, it is not my repo, decisions obviously lie with the team!

[wooorm]

do you want me to submit a PR once done?

Yes please.

despite not being a part of core markdown

This is due to legacy reasons.
Newer things such as math, directives, MDX, don’t have that.

being able to extend PhrasingContent/FlowContent/ListContent would be more conceptually correct

Maybe, yeah. It’s more correct for things like math/directives/MDX etc, true. Not sure how it would work. It gets complex fast. It’s also easy to mess up.