From 744717f6083b278fd704e7a46379a596a2efc91d Mon Sep 17 00:00:00 2001 From: molfar Date: Sat, 19 Jun 2021 13:43:47 +0300 Subject: [PATCH] Meilisearch extra (#316) --- README.md | 1 + docs/_layouts/default.html | 1 + docs/api/backend.md | 2 +- docs/extras.md | 1 + docs/extras/items.md | 9 +- docs/extras/meilisearch.md | 100 ++++++++++++++++++ docs/how-to.md | 4 +- lib/pagy/extras/meilisearch.rb | 55 ++++++++++ pagy.manifest | 1 + tasks/test.rake | 1 + test/mock_helpers/meilisearch.rb | 36 +++++++ test/pagy/extras/items_test.rb | 8 +- test/pagy/extras/meilisearch_test.rb | 103 +++++++++++++++++++ test/pagy/extras/meilisearch_test.rb.rematch | 62 +++++++++++ 14 files changed, 375 insertions(+), 9 deletions(-) create mode 100644 docs/extras/meilisearch.md create mode 100644 lib/pagy/extras/meilisearch.rb create mode 100644 test/mock_helpers/meilisearch.rb create mode 100644 test/pagy/extras/meilisearch_test.rb create mode 100644 test/pagy/extras/meilisearch_test.rb.rematch diff --git a/README.md b/README.md index 2ba9de4db..239c35459 100644 --- a/README.md +++ b/README.md @@ -109,6 +109,7 @@ Use the official extras, or write your own in just a few lines. Extras add speci - [countless](http://ddnexus.github.io/pagy/extras/countless): Paginate without the need of any count, saving one query per rendering - [elasticsearch_rails](http://ddnexus.github.io/pagy/extras/elasticsearch_rails): Paginate `ElasticsearchRails` response objects - [headers](http://ddnexus.github.io/pagy/extras/headers): Add RFC-8288 compliant http response headers (and other helpers) useful for API pagination +- [meilisearch](http://ddnexus.github.io/pagy/extras/meilisearch): Paginate `Meilisearch` results - [metadata](http://ddnexus.github.io/pagy/extras/metadata): Provides the pagination metadata to Javascript frameworks like Vue.js, react.js, etc. - [searchkick](http://ddnexus.github.io/pagy/extras/searchkick): Paginate `Searchkick::Results` objects diff --git a/docs/_layouts/default.html b/docs/_layouts/default.html index 999060216..c246de84f 100644 --- a/docs/_layouts/default.html +++ b/docs/_layouts/default.html @@ -118,6 +118,7 @@

{{ site.title | default: site.github.repository_name }}

Items

Overflow

Materialize

 +

Meilisearch

Metadata

Navs

Searchkick

 diff --git a/docs/api/backend.md b/docs/api/backend.md index 96d4f9d17..ba087582b 100644 --- a/docs/api/backend.md +++ b/docs/api/backend.md @@ -9,7 +9,7 @@ For overriding convenience, the `pagy` method calls two sub-methods that you may **Notice**: Keep in mind that the whole module is basically providing a single functionality: getting a Pagy instance and the paginated items. You could re-write the whole module as one single and simpler method specific to your need, eventually gaining a few IPS in the process. If you seek a bit more performance you are encouraged to [write your own Pagy methods](#writing-your-own-pagy-methods). -Check also the [array](../extras/array.md), [searchkick](../extras/searchkick.md) and [elasticsearch_rails](../extras/elasticsearch_rails.md) extras for specific backend customizations. +Check also the [array](../extras/array.md), [searchkick](../extras/searchkick.md), [elasticsearch_rails](../extras/elasticsearch_rails.md) and [meilisearch](extras/meilisearch.md) extras for specific backend customizations. ## Synopsis diff --git a/docs/extras.md b/docs/extras.md index 2bf2b4d97..25ebf1305 100644 --- a/docs/extras.md +++ b/docs/extras.md @@ -18,6 +18,7 @@ Pagy comes with a few optional extensions/extras: | `i18n` | Use the `I18n` gem instead of the pagy implementation | [i18n.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/i81n.rb), [documentation](extras/i18n.md) | | `items` | Allow the client to request a custom number of items per page with a ready to use selector UI | [items.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/items.rb), [documentation](extras/items.md) | | `materialize` | Add nav, nav_js and combo_nav_js helpers for the Materialize CSS [pagination component](https://materializecss.com/pagination.html) | [materialize.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/materialize.rb), [documentation](extras/materialize.md) | +| `meilisearch` | Paginate `Meilisearch` results efficiently | [meilisearch.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/meilisearch.rb), [documentation](extras/meilisearch.md) | | `metadata` | Provides the pagination metadata to Javascript frameworks like Vue.js, react.js, etc. | [metadata.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/metadata.rb), [documentation](extras/metadata.md) | | `navs` | Add nav_js and combo_nav_js javascript unstyled helpers | [navs.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/navs.rb), [documentation](extras/navs.md) | | `overflow` | Allow for easy handling of overflowing pages | [overflow.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/overflow.rb), [documentation](extras/overflow.md) | diff --git a/docs/extras/items.md b/docs/extras/items.md index a01c33802..5d1537c04 100644 --- a/docs/extras/items.md +++ b/docs/extras/items.md @@ -5,7 +5,7 @@ title: Items Allow the client to request a custom number of items per page with an optional selector UI. It is useful with APIs or user-customizable UIs. -It works also with the [countless](countless.md), [searchkick](searchkick.md) and [elasticsearch_rails](elasticsearch_rails.md) extras. +It works also with the [countless](countless.md), [searchkick](searchkick.md), [elasticsearch_rails](elasticsearch_rails.md) and [meilisearch](extras/meilisearch.md) extras. ## Synopsis @@ -20,9 +20,9 @@ require 'pagy/extras/items' # you can disable it explicitly for specific requests @pagy, @records = pagy(Product.all, enable_items_extra: false) - -# or... - + +# or... + # disable it by default (opt-in) Pagy::VARS[:enable_items_extra] = false # default true # in this case you have to enable it explicitly when you want it @@ -111,4 +111,3 @@ When the items number is changed with the selector, pagy will reload the paginat This method can take an extra `id` argument, which is used to build the `id` attribute of the `nav` tag. Since the internal automatic id generation is based on the code line where you use the helper, you _must_ pass an explicit id if you are going to use more than one `*_js` call in the same line for the same file. **Notice**: passing an explicit id is also a bit faster than having pagy to generate one. - diff --git a/docs/extras/meilisearch.md b/docs/extras/meilisearch.md new file mode 100644 index 000000000..2e3537313 --- /dev/null +++ b/docs/extras/meilisearch.md @@ -0,0 +1,100 @@ +--- +title: Meilisearch +--- +# Meilisearch Extra + +This extra deals with the pagination of `Meilisearch` results either by creating a `Pagy` object out of an (already paginated) `Meilisearch` results or by creating the `Pagy` and `Meilisearch` results from the backend params. + +## Synopsis + +See [extras](../extras.md) for general usage info. + +Require the extra in the `pagy.rb` initializer: + +```ruby +require 'pagy/extras/meilisearch' +``` + +### Passive mode + +If you have an already paginated `Meilisearch` results, you can get the `Pagy` object out of it: + +```ruby +@results = Model.search(nil, offset: 10, limit: 10, ...) +@pagy = Pagy.new_from_meilisearch(@results, ...) +``` + +### Active Mode + +If you want Pagy to control the pagination, getting the page from the params, and returning both the `Pagy` and the Meilisearch results automatically (from the backend params): + +Extend your model: + +```ruby +extend Pagy::Meilisearch +``` + +In a controller use `pagy_search` in place of `search`: + +```ruby +results = Article.pagy_search(params[:q]) +@pagy, @results = pagy_meilisearch(results, items: 10) +``` + +## Files + +- [meilisearch.rb](https://github.com/ddnexus/pagy/blob/master/lib/pagy/extras/meilisearch.rb) + +## Pagy.new_from_meilisearch + +This constructor accepts a Meilisearch as the first argument, plus the usual optional variable hash. It sets the `:items`, `:page` and `:count` pagy variables extracted/calculated out of the Meilisearch object. + +```ruby +@results = Model.search(nil, offset: 10, limit: 10, ...) +@pagy = Pagy.new_from_meilisearch(@results, ...) +``` + +**Notice**: you have to take care of manually manage all the params for your search, however the method extracts the `:items`, `:page` and `:count` from the results object, so you don't need to pass that again. If you prefer to manage the pagination automatically, see below. + +## Pagy::Meilisearch + +Extend your model with the Pagy::Meilisearch` micro-moudule: + +```ruby +extend Pagy::Meilisearch +``` + +The `Pagy::ElasticsearchRails::Search` adds the `pagy_search` class method that you must use in place of the standard `search` method when you want to paginate the search response. + +### pagy_search(...) + +This method accepts the same arguments of the `search` method and you must use it in its place. This extra uses it in order to capture the arguments, automatically merging the calculated `:offset` and `:limit` options before passing them to the standard `search` method internally. + +## Variables + +| Variable | | Description | Default | +|:----------------------------|:-----------------------------------------------|:---------------|:--------| +| `:meilisearch_search_method` | customizable name of the `:pagy_search` method | `:pagy_search` | | + +## Methods + +This extra adds the `pagy_meilisearch` method to the `Pagy::Backend` to be used when you have to paginate a Meilisearch object. It also adds a `pagy_meilisearch_get_vars` sub-method, used for easy customization of variables by overriding. + +### pagy_meilisearch(Model.pagy_search(...), vars={}}) + +This method is similar to the generic `pagy` method, but specialized for Meilisearch. (see the [pagy doc](../api/backend.md#pagycollection-varsnil)) + +It expects to receive a `Model.pagy_search(...)` result and returns a paginated response. You can use it in a couple of ways: + +```ruby +@pagy, @results = pagy_meilisearch(Model.pagy_search(params[:q]), ...) +... +@records = @results.results + +# or directly with the collection you need (e.g. records) +@pagy, @records = pagy_meilisearch(Model.pagy_search(params[:q]).results, ...) +``` + +### pagy_meilisearch_get_vars(array) + +This sub-method is similar to the `pagy_get_vars` sub-method, but it is called only by the `pagy_meilisearch` method. (see the [pagy_get_vars doc](../api/backend.md#pagy_get_varscollection-vars)). diff --git a/docs/how-to.md b/docs/how-to.md index 12bd075cc..e1f6c720b 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -133,7 +133,7 @@ Notice: Older versions run on ruby 1.9+ or jruby 1.7+ till ruby <3.0 Pagy works out of the box in a web app assuming that: - You are using a `Rack` based framework (Rails, Sinatra, Padrino, etc.) -- The collection to paginate is an ORM collection (e.g. ActiveRecord scope) or other collections supported by some backend extra ([array](extras/array.md), [elasticsearch_rails](extras/elasticsearch_rails.md), [searchkick](extras/searchkick.md), ...) +- The collection to paginate is an ORM collection (e.g. ActiveRecord scope) or other collections supported by some backend extra ([array](extras/array.md), [elasticsearch_rails](extras/elasticsearch_rails.md), [searchkick](extras/searchkick.md), [meilisearch](extras/meilisearch.md), ...) - The controller where you include `Pagy::Backend` responds to a `params` method - The view where you include `Pagy::Frontend` responds to a `request` method returning a `Rack::Request` instance. @@ -402,7 +402,7 @@ Ransack `result` returns an `ActiveRecord` collection, which can be paginated ou ## Paginate Elasticsearch results -Pagy has a couple of extras for gems returning elasticsearch results: [elasticsearch_rails](extras/elasticsearch_rails.md) and [searchkick](extras/searchkick.md) +Pagy has a couple of extras for gems returning elasticsearch results: [elasticsearch_rails](extras/elasticsearch_rails.md), [searchkick](extras/searchkick.md) and [meilisearch](extras/meilisearch.md) ## Paginate pre-offsetted and pre-limited collections diff --git a/lib/pagy/extras/meilisearch.rb b/lib/pagy/extras/meilisearch.rb new file mode 100644 index 000000000..66d5b53d8 --- /dev/null +++ b/lib/pagy/extras/meilisearch.rb @@ -0,0 +1,55 @@ +# frozen_string_literal: true + +class Pagy + + VARS[:meilisearch_search_method] ||= :pagy_search + + module Meilisearch + # returns an array used to delay the call of #search + # after the pagination variables are merged to the options + def pagy_meilisearch(term = nil, **vars) + [self, term, vars] + end + alias_method VARS[:meilisearch_search_method], :pagy_meilisearch + end + + # create a Pagy object from a Meilisearch results + def self.new_from_meilisearch(results, vars={}) + vars[:items] = results.raw_answer[:limit] + vars[:page] = [results.raw_answer[:offset] / vars[:items], 1].max + vars[:count] = results.raw_answer[:nbHits] + new(vars) + end + + # Add specialized backend methods to paginate Meilisearch results + module Backend + private + + # Return Pagy object and results + def pagy_meilisearch(pagy_search_args, vars = {}) + model, term, options = pagy_search_args + vars = pagy_meilisearch_get_vars(nil, vars) + options[:limit] = vars[:items] + options[:offset] = (vars[:page] - 1) * vars[:items] + results = model.search(term, **options) + vars[:count] = results.raw_answer[:nbHits] + + pagy = Pagy.new(vars) + # with :last_page overflow we need to re-run the method in order to get the hits + return pagy_meilisearch(pagy_search_args, vars.merge(page: pagy.page)) \ + if defined?(Pagy::UseOverflowExtra) && pagy.overflow? && pagy.vars[:overflow] == :last_page + + [ pagy, results ] + end + + # Sub-method called only by #pagy_meilisearch: here for easy customization of variables by overriding + # the _collection argument is not available when the method is called + def pagy_meilisearch_get_vars(_collection, vars) + pagy_set_items_from_params(vars) if defined?(UseItemsExtra) + vars[:items] ||= VARS[:items] + vars[:page] ||= (params[ vars[:page_param] || VARS[:page_param] ] || 1).to_i + vars + end + + end +end diff --git a/pagy.manifest b/pagy.manifest index a2e72a475..633aa8e66 100644 --- a/pagy.manifest +++ b/pagy.manifest @@ -50,6 +50,7 @@ lib/pagy/extras/headers.rb lib/pagy/extras/i18n.rb lib/pagy/extras/items.rb lib/pagy/extras/materialize.rb +lib/pagy/extras/meilisearch.rb lib/pagy/extras/metadata.rb lib/pagy/extras/navs.rb lib/pagy/extras/overflow.rb diff --git a/tasks/test.rake b/tasks/test.rake index cabd8280d..1340be9d2 100644 --- a/tasks/test.rake +++ b/tasks/test.rake @@ -11,6 +11,7 @@ test_tasks = {} i18n items items_trim + meilisearch overflow searchkick shared_json diff --git a/test/mock_helpers/meilisearch.rb b/test/mock_helpers/meilisearch.rb new file mode 100644 index 000000000..8716411b7 --- /dev/null +++ b/test/mock_helpers/meilisearch.rb @@ -0,0 +1,36 @@ +# frozen_string_literal: true + +require 'pagy/extras/meilisearch' + +module MockMeilisearch + + RESULTS = { 'a' => ('a-1'..'a-1000').to_a, + 'b' => ('b-1'..'b-1000').to_a }.freeze + + class Results < Array + + def initialize(query, params = {}) + @query = query + @params = { offset: 0, limit: 10_000 }.merge(params) + super RESULTS[@query].slice(@params[:offset], @params[:limit]) || [] + end + + def raw_answer + { + hits: self, + offset: @params[:offset], + limit: @params[:limit], + nbHits: RESULTS[@query].length + } + end + end + + class Model + + def self.search(*args) + Results.new(*args) + end + + extend Pagy::Meilisearch + end +end diff --git a/test/pagy/extras/items_test.rb b/test/pagy/extras/items_test.rb index 5f9031b71..f010c1ae3 100644 --- a/test/pagy/extras/items_test.rb +++ b/test/pagy/extras/items_test.rb @@ -3,6 +3,7 @@ require_relative '../../test_helper' require_relative '../../mock_helpers/elasticsearch_rails' require_relative '../../mock_helpers/searchkick' +require_relative '../../mock_helpers/meilisearch' require_relative '../../mock_helpers/arel' require 'pagy/extras/countless' require 'pagy/extras/arel' @@ -18,6 +19,11 @@ def test_items_vars_params(items, vars, params) _(pagy.items).must_equal items _(records.size).must_equal items end + [[:pagy_meilisearch, MockMeilisearch::Model]].each do |meth, mod| + pagy, records = controller.send meth, mod.pagy_search('a'), vars + _(pagy.items).must_equal items + _(records.size).must_equal items + end %i[pagy pagy_countless pagy_array pagy_arel].each do |meth| pagy, records = controller.send meth, @collection, vars _(pagy.items).must_equal items @@ -34,7 +40,7 @@ def test_items_vars_params(items, vars, params) it 'uses the defaults' do vars = {} controller = MockController.new - %i[pagy_elasticsearch_rails_get_vars pagy_searchkick_get_vars].each do |method| + %i[pagy_elasticsearch_rails_get_vars pagy_searchkick_get_vars pagy_meilisearch_get_vars].each do |method| merged = controller.send method, nil, vars _(merged[:items]).must_equal 20 end diff --git a/test/pagy/extras/meilisearch_test.rb b/test/pagy/extras/meilisearch_test.rb new file mode 100644 index 000000000..524bc857f --- /dev/null +++ b/test/pagy/extras/meilisearch_test.rb @@ -0,0 +1,103 @@ +# frozen_string_literal: true + +require_relative '../../test_helper' +require_relative '../../mock_helpers/meilisearch' +require 'pagy/extras/overflow' + +describe 'pagy/extras/meilisearch' do + + describe 'model#pagy_search' do + it 'extends the class with #pagy_search' do + _(MockMeilisearch::Model).must_respond_to :pagy_search + end + it 'returns class and arguments' do + _(MockMeilisearch::Model.pagy_search('a', b:2)).must_equal [MockMeilisearch::Model, 'a', {b: 2}] + end + it 'allows the query argument to be optional' do + _(MockMeilisearch::Model.pagy_search(b:2)).must_equal [MockMeilisearch::Model, nil, {b: 2}] + end + it 'adds an empty option hash' do + _(MockMeilisearch::Model.pagy_search('a')).must_equal [MockMeilisearch::Model, 'a', {}] + end + end + + describe 'controller_methods' do + let(:controller) { MockController.new } + + describe '#pagy_meilisearch' do + before do + @collection = MockCollection.new + end + it 'paginates response with defaults' do + pagy, results = controller.send(:pagy_meilisearch, MockMeilisearch::Model.pagy_search('a')) + _(pagy).must_be_instance_of Pagy + _(pagy.count).must_equal 1000 + _(pagy.items).must_equal Pagy::VARS[:items] + _(pagy.page).must_equal controller.params[:page] + _(results.length).must_equal Pagy::VARS[:items] + _(results).must_rematch + end + it 'paginates with vars' do + pagy, results = controller.send(:pagy_meilisearch, MockMeilisearch::Model.pagy_search('b'), page: 2, items: 10, link_extra: 'X') + _(pagy).must_be_instance_of Pagy + _(pagy.count).must_equal 1000 + _(pagy.items).must_equal 10 + _(pagy.page).must_equal 2 + _(pagy.vars[:link_extra]).must_equal 'X' + _(results.length).must_equal 10 + _(results).must_rematch + end + it 'paginates with overflow' do + pagy, results = controller.send(:pagy_meilisearch, MockMeilisearch::Model.pagy_search('b'), page: 200, items: 10, link_extra: 'X', overflow: :last_page) + _(pagy).must_be_instance_of Pagy + _(pagy.count).must_equal 1000 + _(pagy.items).must_equal 10 + _(pagy.page).must_equal 100 + _(pagy.vars[:link_extra]).must_equal 'X' + _(results.length).must_equal 10 + _(results).must_rematch + end + end + + describe '#pagy_meilisearch_get_vars' do + it 'gets defaults' do + vars = {} + merged = controller.send :pagy_meilisearch_get_vars, nil, vars + _(merged.keys).must_include :page + _(merged.keys).must_include :items + _(merged[:page]).must_equal 3 + _(merged[:items]).must_equal 20 + end + it 'gets vars' do + vars = {page: 2, items: 10, link_extra: 'X'} + merged = controller.send :pagy_meilisearch_get_vars, nil, vars + _(merged.keys).must_include :page + _(merged.keys).must_include :items + _(merged.keys).must_include :link_extra + _(merged[:page]).must_equal 2 + _(merged[:items]).must_equal 10 + _(merged[:link_extra]).must_equal 'X' + end + end + + describe 'Pagy.new_from_meilisearch' do + it 'paginates results with defaults' do + results = MockMeilisearch::Model.search('a') + pagy = Pagy.new_from_meilisearch(results) + _(pagy).must_be_instance_of Pagy + _(pagy.count).must_equal 1000 + _(pagy.items).must_equal 1000 + _(pagy.page).must_equal 1 + end + it 'paginates results with vars' do + results = MockMeilisearch::Model.search('b', limit: 15, offset: 30) + pagy = Pagy.new_from_meilisearch(results, link_extra: 'X') + _(pagy).must_be_instance_of Pagy + _(pagy.count).must_equal 1000 + _(pagy.items).must_equal 15 + _(pagy.page).must_equal 2 + _(pagy.vars[:link_extra]).must_equal 'X' + end + end + end +end diff --git a/test/pagy/extras/meilisearch_test.rb.rematch b/test/pagy/extras/meilisearch_test.rb.rematch new file mode 100644 index 000000000..3847f40e9 --- /dev/null +++ b/test/pagy/extras/meilisearch_test.rb.rematch @@ -0,0 +1,62 @@ +--- +"[1] pagy/extras/meilisearch::controller_methods::#pagy_meilisearch#test_0001_paginates response with defaults": !ruby/array:MockMeilisearch::Results + internal: + - a-41 + - a-42 + - a-43 + - a-44 + - a-45 + - a-46 + - a-47 + - a-48 + - a-49 + - a-50 + - a-51 + - a-52 + - a-53 + - a-54 + - a-55 + - a-56 + - a-57 + - a-58 + - a-59 + - a-60 + ivars: + :@query: a + :@params: + :offset: 40 + :limit: 20 +"[1] pagy/extras/meilisearch::controller_methods::#pagy_meilisearch#test_0002_paginates with vars": !ruby/array:MockMeilisearch::Results + internal: + - b-11 + - b-12 + - b-13 + - b-14 + - b-15 + - b-16 + - b-17 + - b-18 + - b-19 + - b-20 + ivars: + :@query: b + :@params: + :offset: 10 + :limit: 10 +"[1] pagy/extras/meilisearch::controller_methods::#pagy_meilisearch#test_0003_paginates with overflow": !ruby/array:MockMeilisearch::Results + internal: + - b-991 + - b-992 + - b-993 + - b-994 + - b-995 + - b-996 + - b-997 + - b-998 + - b-999 + - b-1000 + ivars: + :@query: b + :@params: + :offset: 990 + :limit: 10