Correct Tokenization

[Sachin04011974]

I am running INCEpTION to get some Named Entities, It is doing really good job of IOB2 tagging of those Named Entities. However it is saying its using StanfordNLP tokenization. However the tokenization with special characters/symbols could have been better.

for e.g.
Token - As Is --> To Be

1. Date - '06', '/', '01', '/', '2019' -> 6/1/2019
2. Email Address - 'xyz', '@', 'mydomain.com' --> [email protected]*
3. US format Telephone Numbers - '123', '-', '456', '-', '7890' --> 123-456-7890*

*True values masked

We tokenized the same text using StanfordNLP Tokenizer and its giving correct tokens (as mentioned under To Be)

Is there a setting through which we achieve the correct tokenization or this is feature not currently incorporated. Please let us know. That will be a great help.

~Sachin

[jcklie]

INCEpTION internally uses the Java Breakiterator when you import documents as plain text. Where did you find information about us using StanfordNLP tokenization? In case you need your own tokenzitation, you need to convert your document into a tokenized file format, e.g. webanno tsv or conll2002. You can get a list of file formats and description in the INCEpTION documentation.

[reckart]

@jcklie Let's close this one and open a separate one for actually making tokenization editable in the brat view?

[jcklie]

@reckart Ok. I created #1778