Stop escaping _ inside words?

[ahamelers]

Question similar to https://github.com/orgs/remarkjs/discussions/885 but for intra-word underscores:

I'm using milkdown, which uses remark-stringify, which uses mdast-util-to-markdown to output markdown.

Is there any way to create an option, using unsafe or handler settings, to stop escaping underscores inside words? My use case involves users with datasets making lists of their files, often using many, many snake_case_file_names. The goal of my project is to output markdown for both rendering as HTML and as a markdown file output. The ultimate markdown renderer has intra-word-emphasis turned off, so escaping the underscores inside words does nothing for me, and it's really very annoying for the plain text markdown file users when they try to reference and copy and paste filenames like snake\_case\_file\_names.

I understand escaping these underscores by default, but it would be ideal if it was something that could be toggled or changed with the options!

[wooorm]

No, escaping things is needed. Imagine JavaScript strings without escaping? That wouldn’t work.

[wooorm]

For file names, I recommend using code: index.js and such

The ultimate markdown

What do you mean by “ultimate”?

intra-word-emphasis

The same is true for CommonMark, assuming underscores.
Perhaps you could investigate/PR when it is safe to use underscores around https://github.com/syntax-tree/mdast-util-to-markdown/blob/fd6a508cc619b862f75b762dcf876c6b8315d330/lib/unsafe.js#L134. Emphasis is complex in CM. They should be same when there are non-whitespace/non-punctuation on both sides

[ahamelers]

No, escaping things is needed.

No one here is suggesting not escaping "things" in general. I am talking about a specific, very common use case, easily regexable, of underscores embedded within words.

For file names, I recommend using code: index.js and such

There is no way I can force hundreds of random users of a WYSIWYG markdown editor to do this.

What do you mean by “ultimate”?

I mean the markdown parser we use to render the markdown as HTML as described:

The goal of my project is to output markdown for both rendering as HTML [...]

[ChristianMurphy]

Welcome @ahamelers! 👋

My use case involves users with datasets making lists of their files, often using many, many snake_case_file_names. The goal of my project is to output markdown for both rendering as HTML and as a markdown file output.

I'm not sure I follow.
It doesn't sound like you need or use remark/milkdown to transform the markdown back to markdown in the first place?
You could display the raw markdown instead.

The ultimate markdown renderer has intra-word-emphasis turned off, so escaping the underscores inside words does nothing for me

I'm not sure I follow this either.
Are you saying there is a different, non-standard markdown parser in the mix?
And that, that parser has different syntax and does not support emphasis?
this documentation would be relevant:

• recommendations around (not) extending markdown https://github.com/micromark/micromark#extending-markdown
• how to extend markdown where needed https://github.com/micromark/micromark#creating-a-micromark-extension
• turning off markdown features https://github.com/micromark/micromark#case-turn-off-constructs

Is there any way to create an option, using unsafe or handler settings, to stop escaping underscores inside words?

Not easily, escapes are needed.

[ahamelers]

Thanks for the welcome!

To try to be more clear, in my system we are encouraging users to create markdown README files describing collections of files (datasets). Because many of our users are not familiar with markdown, we provide a WYSIWYG editor in the browser for this purpose. The markdown output of the editor is used for the dual purpose of creating downloadable files as part of the file collection, and of being parsed for display as HTML (basically, exactly like a README on github; you can see the markdown rendered as HTML on the web or download the raw file).

Are you saying there is a different, non-standard markdown parser in the mix?
And that, that parser has different syntax and does not support emphasis?

We use a standard markdown parser to display the markdown created, just like github does for README files and for these very comments. Settings to turn off intra-word-emphasis are absolutely standard in tons of markdown parsers. Github itself does not support intra-word-emphasis—thus in this comment I can write "something_like_this" without that intra-word _like_ being italicized.

[ChristianMurphy]

Because many of our users are not familiar with markdown, we provide a WYSIWYG editor in the browser for this purpose. The markdown output of the editor is used for the dual purpose of creating downloadable files as part of the file collection, and of being parsed for display as HTML (basically, exactly like a README on github; you can see the markdown rendered as HTML on the web or download the raw file).

Either you trust your users to understand valid CommonMark or you don't.
This feels like trying to have it both ways. 😅
Education on how to read and write Markdown would likely be more effective than trying to shape output in a way that is more likely to create edge cases and issues for said users.

We use a standard markdown parser to display the markdown created, just like github does for README files and for these very comments. Settings to turn off intra-word-emphasis are absolutely standard in tons of markdown parsers. Github itself does not support intra-word-emphasis—thus in this comment I can write "something_like_this" without that intra-word like being italicized.

I am talking about a specific, very common use case, easily regexable, of underscores embedded within words.

I appreciate the sentiment here, but inter-word handling is not a simple and easily regexable problem.
Emphasis in markdown is complex, and has a lot of edge cases. (see https://github.com/syntax-tree/mdast-util-to-markdown/issues?q=is%3Aissue+roundtripping+emphasis+ for some example)
As @wooorm mentions you are welcome to look for improvement to the rules around where escaping is needed.

[ahamelers]

Either you trust your users to understand valid CommonMark or you don't.
This feels like trying to have it both ways. 😅
Education on how to read and write Markdown would likely be more effective than trying to shape output in a way that is more likely to create edge cases and issues for said users.

We've done tons of user testing to find that the people downloading file collections want and appreciate markdown READMEs, while the people creating and sharing file collections don't care or know about markdown and need to be convinced to care about READMEs. Our educational focus is thus on why they should create READMEs at all. If "Either you trust your users to understand valid CommonMark or you don't" is the ethos here, it's hard to understand how there could be a need or a use case for WYSIWYG markdown editors, prosemirror, remark, remark-stringify, and mdast-util-to-markdown at all.

As @wooorm mentions you are welcome to look for improvement to the rules around where escaping is needed.

I have looked at the unsafe settings for _ and attempted to modify the unsafe options sent through remark-stringify, with no results. I do no wish to alter the unsafe settings for everyone, but to feed in an option changing it for my use case. Thus my opening this question! Is the take away here that the escaping of characters, or existing character unsafe settings, are not customizable in mdast-util-to-markdown?

[ChristianMurphy]

If "Either you trust your users to understand valid CommonMark or you don't" is the ethos here, it's hard to understand how there could be a need or a use case for WYSIWYG markdown editors, prosemirror, remark, remark-stringify, and mdast-util-to-markdown at all.

CommonMark is a language. Like with any human or computer language, people need a baseline understanding to work with it.
remark and its surround tools can provide support in working with content, helping users better understand the language, and with efficiency.
Just like people need to understand JavaScript to work with it, yet IDEs, Linters, build tools, and type checkers still exist and add value.

We've done tons of user testing to find that the people downloading file collections want and appreciate markdown READMEs, while the people creating and sharing file collections don't care or know about markdown and need to be convinced to care about READMEs.

Great! So convince them and teach them about markdown.

Is the take away here that the escaping of characters, or existing character unsafe settings, are not customizable in mdast-util-to-markdown?

It is customizable though a syntax extension, as was noted in my previous comment.
You can find example of this by looking at https://github.com/syntax-tree/mdast-util-to-markdown/tree/fd6a508cc619b862f75b762dcf876c6b8315d330#list-of-extensions and searching for unsafe in each.

The take away, is that changing the settings should be done with caution.
There are a lot of edge cases with emphasis which can and will break users' content.
Don't break all your users in a rush to make the code style look a specific way.

[ahamelers]

Thanks, I'll look through the link provided and see what I can come up with. I hope that others who may look at this discussion in the future find relevance in your warnings and definition of the concept of a markup language.

[ChristianMurphy]

Another note worth adding here, there are other remark based serializers beyond remark-stringify, notably https://prettier.io/
Prettier is significantly more opinionated in structuring markdown, which may be good for beginner users, and aims to remove escapes in more spots.