Unescape tokens on md-json conversion

[AMR-KELEG]

fixes #6087
On converting json nlu data into markdown, tokens like: "\n" are
espaced to "\n".
However, on converting markdown nlu data into json,
Unescaping isn't done

Proposed changes:

• ...

Status (please check what you already did):

• added some tests for the functionality
• updated the documentation
• updated the changelog (please check changelog for instructions)
• reformat files using black (please check Readme for instructions)

[sara-tagger]

Thanks for submitting a pull request 🚀 @akelad will take a look at it as soon as possible ✨

[tabergma]

Thanks for taking this over!

Code wise it looks good 🚀 However, could you please add a test here and also add a changelog entry. Thanks.

[tabergma]

Why did you changed this?

[AMR-KELEG]

The item_regex will fail in case of having whitespace tokens in the line since they aren't escaped anymore.

import re

item_regex = re.compile(r"\s*[-*+]\s*((?:.+\s*)*)")
old_item_regex = re.compile(r"\s*[-*+]\s*(.*)")

t = '- THIS IS A MD ENTRY WITH a "\\n" token'
print(repr(t)) 
# '- THIS IS A MD ENTRY WITH a "\\n" token'

unespaced_t = decode_string(t)
print(repr(unespaced_t)) 
# '- THIS IS A MD ENTRY WITH a "\n" token'

re.match(old_item_regex, unespaced_t).groups(0)[0] 
#'THIS IS A MD ENTRY WITH a "'

re.match(item_regex, unespaced_t).groups(0)[0] 
# 'THIS IS A MD ENTRY WITH a "\n" token'

[tabergma]

Great work! 💯

[rasabot]

Could not update branch. Most likely this is due to a merge conflict. Please update the branch manually and fix any issues.

[tabergma]

@AMR-KELEG Can you please resolve the merge conflicts? master is getting updated quite frequently right now, will try to get it in as soon as the conflicts are resolved.

[AMR-KELEG]

@tabergma I have updated the files.
I think that the commit history is a little bit messy,
Should I try to squash the commits somehow?
Or can you at least squash them on merging to master?
I believe the change isn't that big so a single commit can still be meaningful.

[tabergma]

I can squash them on merging into master. Thanks for updating!

[akelad]

@tabergma can we just use the ready-to-merge label here?

[tabergma]

Did not worked the last time I tried. But @tmbo mentioned that he could help merging this.

[tmbo]

I don't think the label works for community PRs (not a 100% sure though).

[AMR-KELEG]

Thanks all for your efforts merging this PR 😅🎉