Make tokenization editable

[jcklie]

Is your feature request related to a problem? Please describe.
When using the builtin tokenizer of INCEpTION, it sometimes does errors. It would be nice if the tokenization can be edited.

Describe the solution you'd like
Make the tokenization somehow editable, e.g. in the brat view.

• #2466 - Displayed Text changes as annotations are added / removed #2474
  • Decoupling of rendering in brat from existence of tokens
  • Option to enable token layer as editable layer
  • Option to enable sentence layer as editable layer
• #1778 - Make tokenization editable #2989
  • Display sentences/tokens in the annotator views on the curation page if they are editable
  • If tokens/sentences are editable, run them through the merge process instead of obtaining them from the template
  • Use a line-oriented editor instead of a sentence-oriented editor while the sentence layer is not read-only
  • Add documentation for editable sentences
  • Disable changing anchoring mode and overlap mode for sentences / tokens
  • Disable coloring rules for sentences
• #1778 - Make tokenization editable #5180
  • Splitting a Token by inserting a zero-width token annotation at the split location
  • Deleting a Token (will resizes adjacent tokens as necessary
  • When deleting a Token at the start/end of a sentence, the merge token does not cross then sentence boundary
  • Deleting a Token can be un-done (via undo functionality)
  • Annotations attached to the token will be appropriately resizes (in particular Lemma)
  • Annotations with token-level granularity will be appropriately resized
• Todo / Open questions
  • Deleting a token also removes attached annotations (eg. POS and transitively dependency relations)
  • What to do with "cross sentence" behavior if a document contains no sentences?
  • Should it be allowed to delete the sentence / token layer?
  • What about tokens / sentences in the curation view?