Unified search query syntax using the full-text search capabilities of the underlying DB

[novocaine]

Support a unified search query syntax which leverages more of the full-text search of each DB we support.

We now support, with the same syntax across Postgresql and Sqlite:

• quoted "search terms"
• AND, OR, - (negation) set operators and parentheses
• Matching words based on their stem, e.g. searches for "dog" matches documents containing "dogs".

This is achieved by

• If on pgsql 11+, pass the user inputer to websearch_to_tsquery
• If on an earlier version of pgsql or sqlite, manually parse the query and transform it into the underlying DB syntax supporting the same query dialect

Multiple terms separated by a space are implicitly ANDed.

Possible Issues:

1. There is no escaping of full-text syntax that might be supported by the DB; e.g. NOT, NEAR, * in sqlite, &, | in pgsql. This runs the risk that people might discover this as accidental functionality and depend on something we don't guarantee.
2. On both pgsql and sqllite, english text is assumed for stemming. To support other languages, it's work: either the target language needs to be known at the time of indexing the message (via room metadata, or otherwise), or it'll be necessary to create a separate index for each language supported. Alternatively, we could just disable stemming for now (but still use the other improvements in this PR).

Before

Prior to this PR, we transformed all full-text queries into a DB full-text query by

1. splitting the query on all non-word characters (i.e. not in the python regex class \w)
2. suffixing all the words with "*" to use prefix matching
3. re-joining the words using '&' as the separator.

For example, the user input foo? bar is transformed into foo* & bar*. This happens for both sqlite and postgres.

This had some drawbacks -

• On sqlite, & isn't actually an operator, its actually just being stripped and then the remaining spaces are interpreted to mean AND.
• Stripping the query of non-word characters means quote search and NOT aren't available to users, despite both engines supporting these operators. Note that client-side seshat search does support these operators as its based on on tantivy
• Prefix search probably isn't what users expect as that's not how search engines usually work; it probably interferes with the expected stemming behaviour; and it might have significant performance impact

Sqlite docs
Postgres docs

And for comparison with how Element clients support search, see the Tantivy docs

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
• Pull request includes a sign off
• Code style is correct
  (run the linters)

Sign-off

Signed-off-by: James Salter [email protected]

[novocaine]

I think it was a bug to pass search_term here rather than search_query..?

[novocaine]

This is currently a no-op, but we will probably want to add stuff here to handle certain cases e.g. #3024

[novocaine]

There's a test failing on sqlite because it relies on some sort of prefixing (searches for labels to match label); this test is passing on postgres because there the database is doing stemming during parsing.

sqlite does actually support stemming, so I'm going to try to enable that to fix (this relies on a db migration, so might get tricky)

[clokep]

@erikjohnston any thoughts on the currently open comment threads?

[erikjohnston]

Otherwise I think its good

[erikjohnston]

Can we move these to multiline strings while we're here 😇

[clokep]

I was planning to do a follow-up PR to update the entire module to multi-line strings. Would that be acceptable?

[erikjohnston]

Sure!