[9.x] Add a DatabaseEngine #564

driesvints · 2021-12-23T11:00:33Z

This PR depends on laravel/framework#40129

This PR adds a new database engine to Scout that's powered by the upcoming whereFulltext addition to Laravel's query builder. Using this, we'll be able to offer natural language search in Scout on fulltext indexes for Scout.

I managed to quickly set this up thanks to the existing Collection engine. We'll probably need to abstract those so functionality is shared.

The way this works is that you define the columns of your index through the toSearchableArray method:

/**
 * Get the indexable data array for the model.
 *
 * @return array
 */
public function toSearchableArray()
{
    return [
        'name' => $this->name,
        'bio' => $this->bio,
    ];
}

Then use it as usual:

$users = User::search('Hello world!')->get();

where statements to limit results are also supported:

$invoices = Invoice::search('Forge')->where('status', 'open')->get();

Later on when whereFulltext gets supported by more database engines in Laravel, the upgrade will be seamless.

Todo

Figure out a better way to configure columns when models have multiple fulltext indexes.
Figure out a way to pass options to the whereFulltext method. Maybe through the scoutMetadata method?
Abstract similar functionality from the collection driver.
Add tests

taylorotwell · 2021-12-23T15:25:43Z

We will want a way to also search on normal columns using LIKE or just normal value check like against IDs, etc.

Inspiration for this can be taken from Nova's database search functionality which builds the or where clauses for you, etc.

tpetry · 2022-01-03T07:09:43Z

We will want a way to also search on normal columns using LIKE or just normal value check like against IDs, etc.

How to decide which one to use? Fulltext search and checking against IDs can be index supported, wildcard search with a leading wildcard can not use an index (except on PostgreSQL with pg_trgm). But wildcard search with a trailing wildcard can use an index, but again only if it is indexed and only for an index which will support the operation (hash indexes won't work).

And even for fulltext search scout has to know which columns are fulltext indexed or the query will be stopped with an error on missing index (MySQL) or will be slow (PostgreSQL).

driesvints · 2022-01-04T13:47:21Z

We will want a way to also search on normal columns using LIKE or just normal value check like against IDs, etc.

@taylorotwell normal value checks are already supported:

$invoices = Invoice::search('Forge')->where('id', 5)->get();

For LIKE they can already use the callback with the search method:

use Laravel\Scout\Builder;

$invoices = Invoice::search('Forge', function ($query, Builder $builder string $search) {
    $query->where('title', 'LIKE', $search);
})->get();

That'll invoke the fulltext search and an additional where like clause.

We could also add a new whereLike method to the Scout builder that only works on the DatabaseEngine:

$invoices = Invoice::search('Forge')->whereLike('title', 'Forge')->get();

But that might be a bit too much and confusion for the other engines?

@taylorotwell I actually want to know more why you want to support LIKE and additional column clauses. It seems to me we need to focus this all towards fulltext indexes? Like @tpetry says, I don't think it's ideal to mix things up with LIKE, etc.

taylorotwell · 2022-01-04T22:39:13Z

@driesvints it's extremely common to search by primary ID in business applications (for example, order ID) in addition to fields that may be full text indexed like a shipping address or order notes field using a single search input field.

So, for example, if I type "1" in the search box I may want to find the user with an ID of 1, while I can also type "Arkansas" in the search box to find orders with "Arkansas" in the full-text indexed notes. Without that functionality I frankly would not find a database driver for Scout useful at all.

The type of search functionality I'm proposing is exactly the kind of search functionality offered by the search field in Laravel Nova, which I think I noted that code may provide some good inspiration here as it works really well across multiple database drivers.

I will look into this PR more tomorrow and will likely work on the implementation if I can.

src/Engines/DatabaseEngine.php

driesvints · 2022-01-06T11:28:23Z

So, for example, if I type "1" in the search box I may want to find the user with an ID of 1, while I can also type "Arkansas" in the search box to find orders with "Arkansas" in the full-text indexed notes. Without that functionality I frankly would not find a database driver for Scout useful at all.

How will you know which field(s) to search? Sounds more like the ID is part of a separate filter than the search box?

The type of search functionality I'm proposing is exactly the kind of search functionality offered by the search field in Laravel Nova, which I think I noted that code may provide some good inspiration here as it works really well across multiple database drivers.

I'll investigate that.

tpetry · 2022-01-06T12:32:09Z

So, for example, if I type "1" in the search box I may want to find the user with an ID of 1, while I can also type "Arkansas" in the search box to find orders with "Arkansas" in the full-text indexed notes. Without that functionality I frankly would not find a database driver for Scout useful at all.

I like that approach but had no idea until now how to do it.

A possible solution for the database driver could be a specification for every column. I like the approach of pmatseykanets/laravel-scout-postgres. Additionally to the normal scout configuration, another array is defined with options for the driver. The search method would be defined for every column (wildcard/fulltext) and be used by the driver to build the query.

Either way a configuration method on the model needs to be supported to pass the $options configuration to whereFulltext.

driesvints · 2022-01-06T12:38:13Z

A possible solution for the database driver could be a specification for every column. I like the approach of pmatseykanets/laravel-scout-postgres. Additionally to the normal scout configuration, another array is defined with options for the driver. The search method would be defined for every column (wildcard/fulltext) and be used by the driver to build the query.

Does that mean that we add "or" clauses for every mapped column?

tpetry · 2022-01-06T18:46:47Z

Does that mean that we add "or" clauses for every mapped column?

That was my first idea. But it does not work as defining multiple columns for fulltext searching would try to do a fulltext search on each by it's own and then or the results, which will not work.

Maybe something like this (i don't like the structure, just sharing the idea):

public function searchableOptions()
{
    return [
        'columns' => [
            'like_search' => ['id', 'email'], // these can be "or"ed
            'fulltext_search' => [ // each group needs to use one whereFullText and then can be "or"ed
                ['body', 'title],
                ['subject', 'description],
            ],
        ],
    ];
}

taylorotwell · 2022-01-06T21:23:25Z

Hey all,

I've pushed my work for today which is a pretty big rewrite of this. It now behaves more like Laravel Nova search with the added ability for real full-text searching with the new whereFullText methods as well as a combination of LIKE and also LIKE with no leading %.

I built a small search interface for myself to test it with:

Overview

So, given the following toSearchableArray method:

The following naive, LIKE based query would be executed by default, allowing very basic searching across all columns:

This is exactly like Nova. Note - this portion of the query is grouped within parenthesis, again, like Nova, for proper logical grouping if the developer adds additional where clauses to the query.

If you try to search by an integer and the primary key type is an integer for the model, it will attempt an exact search against that value instead of a LIKE - again, just like Nova.

Full Text Support

You may easily specify which columns actually have full text indexes using Attributes:

This now generates a query like so:

LIKE Without Wildcard Prefix

Some columns you may want to be searched using where like 'foo%' - with no leading % on the LIKE for better performance. This is also supported:

Other Improvements

I have made quite a few other improvements. Most notably, the original implementation of this driver executed 2 database queries... one to perform the search and then an extra, extraneous query on all the matching models by ID using a where in statement. That has been removed and now the models returned from the initial search query are used as the actual result set so only 1 query is executed to perform the search.

I've also updated the pagination logic to handle the LIMIT and OFFSET at the database level instead of potentially returning thousands of models to Laravel and doing it via the collections.

Caveats

The Scout shouldBeSearchable method is sort of inherently not compatible with a database engine that searches your tables directly. The purpose of that method was to prevent models from being added to search indexes at all. This makes sense for things where the index is separate from the underlying table like Algolia, but for a database engine, the data is always in the table.

Therefore, when using the database engine users would be encouraged / required to add the proper where constraints to the query when searching:

Post::search('foo')->where('published', true)->get();

Otherwise, we would be required to retrieve every matching model into memory (even when paginating) and manually invoke shouldBeSearchable on that model, which isn't practical for thousands of models.

robsontenorio · 2022-01-07T03:05:11Z

Does fulltext approach , instead MeiliSeach or ElasticSearch, implies in too much heavy load on database? Locks ? Performance?

Does anybody has any concerns about that ?

I really would love to get rid off that extra piece of infrastructure (Melisearch/Elastic)

fhferreira · 2022-01-07T04:31:37Z

Does fulltext approach , instead MeiliSeach or ElasticSearch, implies in too much heavy load on database? Locks ? Performance?

Does anybody has any concerns about that ?

I really would love to get rid off that extra piece of infrastructure (Melisearch/Elastic)

+1

driesvints · 2022-01-07T07:58:11Z

@taylorotwell this looks amazing, thanks! I do believe the SearchUsingFullText attribute needs an options array for extra options to pass to the whereFullText method for engines underneath to use.

Most likely we'll need to provide a way to map those options separate per column.

/cc @tpetry ☝️

crynobone · 2022-01-07T10:32:26Z

src/Engines/DatabaseEngine.php

+                    }
+
+                    $query->orWhere(
+                        $builder->model->qualifyColumn($column),


Suggested change

$builder->model->qualifyColumn($column),

$query->qualifyColumn($column),

crynobone · 2022-01-07T10:32:36Z

src/Engines/DatabaseEngine.php

+            foreach ($columns as $column) {
+                if (in_array($column, $fullTextColumns)) {
+                    $query->orWhereFullText(
+                        $builder->model->qualifyColumn($column),


Suggested change

$builder->model->qualifyColumn($column),

$query->qualifyColumn($column),

Available since 5.5 laravel/framework#22577

taylorotwell · 2022-01-07T15:56:51Z

Does fulltext approach , instead MeiliSeach or ElasticSearch, implies in too much heavy load on database? Locks ? Performance?

Does anybody has any concerns about that ?

I really would love to get rid off that extra piece of infrastructure (Melisearch/Elastic)

I have no idea on the performance. Depends on the size of your database I'm sure. This driver is intended for usage on smaller data sets and workloads where such performance isn't a concern.

tpetry · 2022-01-07T16:31:10Z

Does fulltext approach , instead MeiliSeach or ElasticSearch, implies in too much heavy load on database? Locks ? Performance?

Does anybody has any concerns about that ?

I really would love to get rid off that extra piece of infrastructure (Melisearch/Elastic)

It's highly dependent on your database, your data model and your queries. In the past MySQL was pretty bad when you did a fulltext search with a condition to filter rows because MySQL was not able to use multiple indexes for a single query. But that has changed with 8.0, whether it will use multiple indexes or not is still depending on a lot of things and some magic of MySQL's query planner. And it's still not choosing multiple indexes today very often, but it's getting better and better.

I only have recent experience with PostgreSQL's fulltext search which is working really nice with a few million entries. And as PostgreSQL will use multiple indexes since 8.1 (2005) the query planner is really good at it and you can blindly throw queries at it and it will execute the fulltext queries very fast. Combined with tight control of word stemming, relevances for columns and ranking for sorting it's a good simpler alternative to Melisearch/Elastic for low/medium workloads. But sure, a single database server can't beat a 6 node Elastic cluster.

As said, there is no general statement to performance, it depends on a lot of things. It's best you try your workload and if it's working you can be happy to remove Melisearch/Elastic from your stack.

--

I just wanted to add that everyone is concerned about "performance". Everyone is concerned about slow databases queries. Databases can do a lot more things efficiently than most people believe. Instead of researching performance problems of solution x just implement it and test it. Nobody is able to predict special behaviour accurately (except if you do stupid things). And everything is always completely dependent on your data model, data size and querying behaviour. Something working perfectly for me may work horrible for you because these 3 properties are different for you - but most often only the first and last one make a difference.

taylorotwell · 2022-01-07T17:36:44Z

Instead of researching performance problems of solution x just implement it and test it.

✔️

driesvints and others added 2 commits December 23, 2021 12:00

Add a DatabaseEngine

571c499

Apply fixes from StyleCI

a9b6630

driesvints changed the title ~~Add a DatabaseEngine~~ [9.x] Add a DatabaseEngine Dec 23, 2021

driesvints requested a review from taylorotwell January 4, 2022 13:47

taylorotwell reviewed Jan 5, 2022

View reviewed changes

src/Engines/DatabaseEngine.php Outdated Show resolved Hide resolved

taylorotwell added 2 commits January 5, 2022 13:40

fix filter

460aaa6

work on database engine

5c2f5a1

Fix filter

adc43c1

taylorotwell added 4 commits January 6, 2022 13:31

performance improvements - formatting

663fe51

fulltext support

3a832ba

add files

22e7f87

fix conflicts

c67ace7

taylorotwell marked this pull request as ready for review January 6, 2022 21:40

taylorotwell added 4 commits January 6, 2022 16:30

fix pagination

d48b3c9

simplify method

a164255

remove temporary variable

be09398

adjust how primary key is searched

4c92dfd

crynobone reviewed Jan 7, 2022

View reviewed changes

taylorotwell and others added 2 commits January 7, 2022 10:07

add order by

6b9f33a

Apply fixes from StyleCI

1ad1ef7

taylorotwell added 3 commits January 8, 2022 11:48

efficient simple pagination

8b4089f

add basic tests

fd7d246

adjust for php 8

4f18e4b

taylorotwell merged commit 0ff8f08 into 9.x Jan 8, 2022

taylorotwell deleted the database-driver branch January 8, 2022 18:09

LastDragon-ru mentioned this pull request Oct 29, 2022

@searchBy/@sortBy optimizations LastDragon-ru/lara-asp#21

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[9.x] Add a DatabaseEngine #564

[9.x] Add a DatabaseEngine #564

driesvints commented Dec 23, 2021 •

edited

Loading

taylorotwell commented Dec 23, 2021

tpetry commented Jan 3, 2022 •

edited

Loading

driesvints commented Jan 4, 2022

taylorotwell commented Jan 4, 2022 •

edited

Loading

driesvints commented Jan 6, 2022

tpetry commented Jan 6, 2022 •

edited

Loading

driesvints commented Jan 6, 2022

tpetry commented Jan 6, 2022

taylorotwell commented Jan 6, 2022 •

edited

Loading

robsontenorio commented Jan 7, 2022

fhferreira commented Jan 7, 2022 •

edited

Loading

driesvints commented Jan 7, 2022 •

edited

Loading

crynobone Jan 7, 2022 •

edited

Loading

crynobone Jan 7, 2022 •

edited

Loading

taylorotwell commented Jan 7, 2022 •

edited

Loading

tpetry commented Jan 7, 2022 •

edited

Loading

taylorotwell commented Jan 7, 2022

	$builder->model->qualifyColumn($column),
	$query->qualifyColumn($column),

[9.x] Add a DatabaseEngine #564

[9.x] Add a DatabaseEngine #564

Conversation

driesvints commented Dec 23, 2021 • edited Loading

Todo

taylorotwell commented Dec 23, 2021

tpetry commented Jan 3, 2022 • edited Loading

driesvints commented Jan 4, 2022

taylorotwell commented Jan 4, 2022 • edited Loading

driesvints commented Jan 6, 2022

tpetry commented Jan 6, 2022 • edited Loading

driesvints commented Jan 6, 2022

tpetry commented Jan 6, 2022

taylorotwell commented Jan 6, 2022 • edited Loading

Overview

Full Text Support

LIKE Without Wildcard Prefix

Other Improvements

Caveats

robsontenorio commented Jan 7, 2022

fhferreira commented Jan 7, 2022 • edited Loading

driesvints commented Jan 7, 2022 • edited Loading

crynobone Jan 7, 2022 • edited Loading

Choose a reason for hiding this comment

crynobone Jan 7, 2022 • edited Loading

Choose a reason for hiding this comment

taylorotwell commented Jan 7, 2022 • edited Loading

tpetry commented Jan 7, 2022 • edited Loading

taylorotwell commented Jan 7, 2022

driesvints commented Dec 23, 2021 •

edited

Loading

tpetry commented Jan 3, 2022 •

edited

Loading

taylorotwell commented Jan 4, 2022 •

edited

Loading

tpetry commented Jan 6, 2022 •

edited

Loading

taylorotwell commented Jan 6, 2022 •

edited

Loading

fhferreira commented Jan 7, 2022 •

edited

Loading

driesvints commented Jan 7, 2022 •

edited

Loading

crynobone Jan 7, 2022 •

edited

Loading

crynobone Jan 7, 2022 •

edited

Loading

taylorotwell commented Jan 7, 2022 •

edited

Loading

tpetry commented Jan 7, 2022 •

edited

Loading