Whitespace Tokenizer incorrectly removes vowel signs for text in Hindi

[dakshvar22]

Rasa version: master

Python version: 3.6.5

Operating system (windows, osx, ...): osx

Issue:
WhitespaceTokenizer.tokenize tries to remove punctuations/symbols using a regex -

words = re.sub(
            # there is a space or an end of a string after it
            r"[^\w#@&]+(?=\s|$)|"
            # there is a space or beginning of a string before it
            # not followed by a number
            r"(\s|^)[^\w#@&]+(?=[^0-9\s])|"
            # not in between numbers and not . or @ or & or - or #
            # e.g. 10'000.00 or [email protected]
            # and not url characters
            r"(?<=[^0-9\s])[^\w._~:/?#\[\]()@!$&*+,;=-]+(?=[^0-9\s])",
            " ",
            text,
        ).split()

This regex cleans up all vowel signs which is wrong.

Example input: 50 क्या आपके पास डेरी मिल्क 10 वाले बॉक्स मिल सकते है
Ideally, the whitespace tokenizer should give the following list of tokens:

['50', 'क्या', 'आपके', 'पास', 'डेरी', 'मिल्क', '10', 'वाले', 'बॉक्स', 'मिल', 'सकते', 'है']

instead, the tokens returned are:

['50', 'क', 'य', 'आपक', 'प', 'स', 'ड', 'र', 'म', 'ल', 'क', '10', 'व', 'ल', 'ब', 'क', 'स', 'म', 'ल', 'सकत', 'ह']

This also leads to misaligned entity annotations as described here.

[SreenijaK]

Hi,
Any update on this issue?

[dakshvar22]

Hi @SreenijaK, unfortunately no we haven't had the bandwidth to get to this. Hopefully by end of next week I should be able to tackle it. In the meantime, if you want to take a jab at it please feel free to do so. :)

[dakshvar22]

Hi @SreenijaK This is now fixed in the latest version of Rasa Open Source. Let us know if this works for you.