Support for Markdown escaping

[martinvonwittich]

Description:

Most Markdown engines support escaping; that is, you can tell them to ignore Markdown formatting by putting a backslash before Markdown characters. For example in Github Flavoured Markdown: https://github.github.com/gfm/#backslash-escapes

This seems to be missing in the Rocket.Chat Markdown engine. It would be especially useful for web integrations that post messages in Rocket.Chat. For example, we have written an integration that automatically posts Redmine links when someone mentions an issue ID like #1234; the integrations formats these links with Markdown, e.g.:

[Project - Error RocketChat/feature-requests#783: Subject](https://redmine.example.com/issues/1234)

Without quoting, special characters in the subject like ] prevent Rocket.Chat from properly rendering the link.

Server Setup Information:

• Version of Rocket.Chat Server: 0.59.3
• Operating System: Debian GNU/Linux 9.2 (stretch)
• Deployment Method(snap/docker/tar/etc): tar
• Number of Running Instances: 1
• DB Replicaset Oplog: Disabled
• Node Version: v4.5.0

Steps to Reproduce:

Attempt to post special Markdown characters that are escaped with a backslash. For example, try to post a text containing asterisks, and prepend a backslash to prevent Rocket.Chat from interpreting this as "bold":

\*This is a test\*

Or try to post links that contains a verbatim ] character:

[Project - Error RocketChat/feature-requests#783: Subject \]](https://redmine.example.com/issues/1234)

[Project - Error RocketChat/feature-requests#783: Subject \[test\]](https://redmine.example.com/issues/1234)

Expected behavior:

Rocket.Chat should display the first message like this (that is, verbatim asterisk, followed by test, followed by verbatim asterisk):

*test*

The links should be displayed like this:

Project - Error RocketChat/feature-requests#783: Subject ]

Project - Error RocketChat/feature-requests#783: Subject [test]

Actual behavior:

Rocket.Chat displays the first message like this:

\*test\*

And the links like this:

[Project - Error RocketChat/feature-requests#783: Subject \]](https://redmine.example.com/issues/1234)

Project - Error RocketChat/feature-requests#783: Subject =!=IfJbf9a34tKf2QEcg=!=

I don't even have the slightest idea what's happening with the second link, Rocket.Chat seems to replace the text in braces with a random string?! It kinda looks like a password, so I've replaced it with a randomly generated similar string just to be sure.

[martinvonwittich]

The last issue I mentied, the fact that RocketChat replaces text in backslashed braces with random password-like strings, also occurs in code blocks:

```
\[test\]
```

is displayed by RocketChat like this:

=!=z63FjnFNp8wqX5zMh=!=

instead of like this:

\[test\]

[nickbroon]

An example of this not rendering properly for escaped braces is here: https://open.rocket.chat/channel/support?msg=qQmNsdpvnqWAtSurC

I would suggest this is a bug, as there is no work around, and not an improvement request.

[1100101]

Regarding brackets within a Markdown URL definition, e.g.:
[Project - Error RocketChat/feature-requests#783: Subject [test]](https://redmine.example.com/issues/1234)
Manual escaping of the verbatim brackets by the user should not be necessary.
Instead, RocketChat's Markdown implementation should figure this out automatically.

The above URL should become:
Project - Error RocketChat/feature-requests#783: Subject [test]

[philn16]

I haven't figured out a workaround for KaTeX escaping the dollar sign for symbolic math as in below:

echo $dollar $hey

[MatthieuPERIN]

Same issue when needing to talk about *.xml & *.xsd files in a chat the rendering brings ".xml &" as bold, please provide a solution !

[localguru]

Something like *text* seems not possible. Here on github I can escape with \*text\* .

[Yolgie]

Basic escaping would be awesome for all the formatting characters `*~_

[kdre]

I also would appreciate if basic escaping would be integrated.

[flvani]

I also would appreciate if basic escaping would be integrated.

[JonathanCabezas]

I also would appreciate....... Wait isn't such a feature obvious in 2020 ? Some devs I swear...

[ApolloLV]

We have a special issue with this. We use Rocket.chat in a german community that uses asterisks in their day-to-day communication ("gendern"). As you can imagine, this is very annoying.

[ApolloLV]

It appears as if it all boils down to a single line doing unconditional replacing in a string:

for (const { token, text, noHtml } of tokenList) {
    if (message.html.indexOf(token) >= 0) {
        message.html = message.html.replace(token, () => (useHtml ? text : noHtml)); // Uses lambda so doesn't need to escape $
    } else {
        missingTokens.push({ token, text, noHtml });
    }
}

Source

[Karasuni]

There is no escaping of special characters in RocketChat?

[lorenzo-w]

Has this issue seriously been unsolved for more than 3 years now? WHAT KIND OF DEVELOPER would even introduce a formatting feature via special characters and not implement a way to escape them in the first place?? And how is the leading open source chat solution incapable of fixing such a simple problem?
I really need answers here, because I just spent weeks moving my German organization to Rocket.Chat and now I have to tell them they can never use an asterisk in their messages ever again (which we actually need frequently)...

[antgel]

You must be new around here, I opened #8332 in 2017, and it's gone to the usual graveyard. RC has historically suffered from a lack of GitHub Issue hygiene. Often someone will come along, set some tags and milestones, then... In fact, I still have seven open issues, opened between 2016 and 2018.

I used to try to explain to the team why it is critical for the health of the project, but it's their project and it's not in their culture / DNA, it's code code code (which is fine but ~2300 open issues?!?!).

Having said that, RC is open source, you're presumably not paying for an Enterprise subscription, so you presumably know what sort of warranty you get. I'm sure your capable, better-developer patches are more than welcome.

[lorenzo-w]

Having said that, RC is open source, you're presumably not paying for an Enterprise subscription, so you presumably know what sort of warranty you get. I'm sure your capable, better-developer patches are more than welcome.

@antgel
Shots fired. In both directions...
Having said that: If I actually did that, do you think they would even merge my patch? After what you said I am not so hopeful about that.

[bernb]

I fully support of the urgency of the topic. As others pointed out, the asterisk has become a de-facto standard in the German language within many corporate and governmental contexts.

Following this issue, I also fully support keeping any communication within issues technical and professional.

[antgel]

Having said that, RC is open source, you're presumably not paying for an Enterprise subscription, so you presumably know what sort of warranty you get. I'm sure your capable, better-developer patches are more than welcome.

@antgel
Shots fired. In both directions...
Having said that: If I actually did that, do you think they would even merge my patch? After what you said I am not so hopeful about that.

Completely understandable, given the 528 currently open PRs. I guess that's partly why there are ~6800(!) forks. Sigh.

[ggazzo]

Hey guys, sorry for the late reply. What can I say? We are fully aware of markdown issues.

I know it's not a good reason for this delay, but the decisions we made and how we implemented our interpreter were terrible, and in a way it became almost impossible to maintain. So we introduced marked as a second option (it was another bad decision as we now have 2 different behaviors and 2 codes to look after).

To solve this, we start a new parser (we're going to remove the other two and not have 3) .

You can check it here: https://github.com/RocketChat/Rocket.Chat.Fuselage/tree/develop/packages/message-parser

The way this parser was built is based on a grammar, the expressiveness is much more powerful, so we are fully capable of making the changes you are asking us to make. The parser already works in the new versions but it is in beta, it doesn't provide ALL the desirable ones yet, but we are very proud with the results. And we intend to give full maintenance to this library.

[antgel]

@ggazzo Thanks for the reply and the transparency. Any clues when we might see this in releases?

More generally, if the team would clean up the issues and pull requests and devote x% of the time to keeping them clean, I think there would be less frustration and users would feel more empowered instead of disenchanted.

It's like technical debt, if you don't pay it back, you grind to a halt. From Agile Principles:

Continuous attention to technical excellence and good design enhances agility.

[zdumitru]

@antgel It's planned to be released after version 3.19. You can already enable new parser now in Admin accounts settings. But don't be too excited, it has so many bugs that I have disabled it right away. Most of our integrations and bot messages that we built look broken or ugly with it. So it's another thing that will take 1-2 years to make work properly.

[bernb]

Will you accept PRs that allow escaping and updates the current code base, i.e. one that is not related to the new parser?
Just asking so it's clear, as you already have that ongoing as a separate sub-project.

[ggazzo]

@antgel It's planned to be released after version 3.19. You can already enable new parser now in Admin accounts settings. But don't be too excited, it has so many bugs that I have disabled it right away. Most of our integrations and bot messages that we built look broken or ugly with it. So it's another thing that will take 1-2 years to make work properly.

@zdumitru could you provide some examples of broken integrations?
we are trying to guarantee the change will be smooth as possible. Actually we are just waiting, mobile implementation

thanks again for the patience and dont give up.

More generally, if the team would clean up the issues and pull requests and devote x% of the time to keeping them clean, I think there would be less frustration and users would feel more empowered instead of disenchanted.

@antgel you are 100% right, we're setting some goals and it's a little tricky to fit the technical debt when we want a bunch of new features "what's working is working" (with some bugs, but ok).

But I swear I am extremely in agreement with your position and certainly one of my personal goals is to reduce our technical debt (sometimes I lack more time, but I never forget). Thank you for your constructive attitude and for anyone else who provides information and shows interest in helping ❤️

[joswhite]

TL;DR:

RocketChat fixed this in 4.0.0 (see the "new message parser" improvement, which closes this issue). To fix it you not only need to upgrade, but also change the message Parser from "General" to "Marked". After we did both of these markdown escaping works fine.