Add null byte as hard context separator

[LukasKalbertodt]

Pull Request

Related issue

Fixes https://github.com/orgs/meilisearch/discussions/744

What does this PR do?

Adds \0 as context separator.

This allows one to use \0 as artificial separator, for example when concatting lots of small strings into a large string. See this discussion for context: https://github.com/orgs/meilisearch/discussions/744

PR checklist

Please check if your PR fulfills the following requirements:

• Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
• Have you read the contributing guidelines?
• Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!

[ManyTheFish]

Hello @LukasKalbertodt,
Thank you for your PR,
I just let you know that the CONTEXT_SEPARATOR list is a promoting list, which means that if we categorize a token as a separator AND this token is part of the CONTEXT_SEPARATOR list, then it will be considered a hard separator.
However, if the token is not categorized as a separator first, it will never be promoted, even if it is part of the CONTEXT_SEPARATOR list.
If you want to consider \0 as a separator by default, you have to add it to the DEFAULT_SEPARATOR list as well!

[ManyTheFish]

Approved, thank you for the contribution!

bors merge

[meili-bors]

Build succeeded:

• Run Clippy
• Run Rustfmt
• tests