LUIS.ai tokenization

[bkmeneguello]

rasa NLU version (Don't know? https://goo.gl/g9QQg2):
0.8.0a3

Used backend (mitie, spacy_sklearn, ...) & plattform (windows, osx, ...):
mitie_sklearn
Linux

Issue:
Although the warning message when LUIS fformat is used, the tokenization is misbehaving in a sensitive way. At luis data format, the startPos and endPos are character indices, but "load_luis_data" treats them as token indices. I think the "load_luis_data" could take its "start" and "end" attributes directly from raw text and then it even shouldn't need to be tokenized.

text = s.get("text")
intent = s.get("intent")
entities = []
for e in s.get("entities") or []:
    start, end = e["startPos"], e["endPos"] + 1
    val = text[start:end]
    entities.append({"entity": e["entity"], "value": val, "start": start, "end": end})

[amn41]

thanks for creating this issue @bkmeneguello !

This is a really good catch: it looks like LUIS have changed the format of their json exports (for the better!)

They used to provide token indices, which were somewhat ambiguous. We will have to update rasa NLU accordingly