Does Meili officially support control characters inside strings?

[LukasKalbertodt]

I tested this (with Meili v1.4.2):

curl \
    -X POST 'http://localhost:7700/indexes/test/documents' \
    -H 'Content-Type: application/json' \
    --data-binary '[{"id": 123, "desc": "foo\u0000bar" } ]'

And it works. I can also retrieve the document again, with the null byte still being inside the desc field. I can even search for foobar and get a match start: 0, length: 7 inside desc.

My question is: does this work by accident or can I rely on Meili working with control characters inside strings?

And a follow up question: will this slow down Meili? Since deserializing JSON with escape codes means that a new string has to be allocated as one can't simply reference a part of the input file.

EDIT: though it seems like Meili treats \0 as a normal word character. For example with query "bar", the document is not found (due to Meili performing prefix search). All control characters should probably be treated the same as ". ", i.e. a strong separator token. Would such a change be welcome in Meili?

[ManyTheFish]

Hello @LukasKalbertodt,

Yes, the control characters are ignored during the tokenization process, meaning that the search never sees any of these characters.
However, the original document is not modified, that's why they are still in the returned documents.

[LukasKalbertodt]

I think it would make sense to treat control characters as "hard separators", like ; for example. Would you accept a PR changing this?

[ManyTheFish]

considering them hard separators means that the separator splits 2 "unrelated" contexts, like sentences. This would remove the possibility of finding your document using the query foobar.
I am not sure that you want this. Am I wrong?

[LukasKalbertodt]

That's indeed what I want. Just talking about the null byte now, I can't imagine there being a use for having null bytes that do not act as hard separators. And classifying the null byte as such would allow me to concat a bunch of unrelated texts together.

For some context: each document in my search index has a large number of unrelated texts. I want Meili to search through all of those texts. I could store them as big array of strings, but since these are thousands, if not more, per document, this slows down Meili. My experiments showed that concatenating them into one big string is way faster (understandably). For that I just need a good separator. Since all these text are unrelated, I specifically do not want to find a document with the texts ["foo", "bar"] via the query "foobar". Right now I'm using ; as separator, which I think works exactly as I want. But it would be even nicer to have a more artificial separator for this.

[ManyTheFish]

Ok, I understand now.
I would accept to categorize the \0 as a hard separator in the Tokenizer,
feel free to open a PR. 😄

[LukasKalbertodt]

Great! meilisearch/charabia#295