fix: make ExtractiveReader handle situations where token_to_chars returns None instead of a (start, end) tuple

[ZanSara]

Related Issues

• fixes ExtractiveReader fails with proper input Documents #6098

Proposed Changes:

Some tokenizers seems to have their token_to_chars function returning None on some inputs, while the function should always return a tuple as long as the token is in the context, according to its own docstring.

Considering that this is likely a bug of the tokenizer, and to prevent such issues from crashing the reader, I propose to use None as start/end positions when this bug occurs. This choice means that if the starting token points to position None, it will match the start of the document, and if the end token points to position None, it will match the end of the document. Such occurrences are logged.

Alternatively we could raise an error message, but the source of this behavior is unclear and likely tokenizer-dependent, see huggingface/transformers#1662, huggingface/transformers#8209.

How did you test it?

Added a new unit test.

Notes for the reviewer

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[julian-risch]

LGTM to prevent the reader from crashing. I am bit concerned that we don't have a better understanding why the issues with the tokenizers occur. It's outside of the Haystack code base, so no issue with the reader, which is good.