Search result pinning

[markharwood]

Result pinning will be a licensed feature of Elasticsearch (not OSS).

Background and requirements

Sometimes it is desirable to artificially boost certain search results for a query and arrange them on page 1 of the search results. To frame the discussion let's start with an outline of a typical search UI that has result pinning:

This is about making ranking fixes, not serving Ads...

Advertisements that are triggered off keyword searches typically appear above organic results (eg when searching Google for product X you’ll typically see competitor ads). In that scenario the promoted content does not typically share the same index/schema as the organic results so does not respond to filtering or sorting selections. The ads are seen as separate content and can be held in independent stores and queried using separate requests.

In contrast, this proposal is about promoting selected documents from the same index that holds the organic results. It is about fixing ranking issues. The artificially placed documents should blend in naturally with organic results so they look the same, are highlighted the same, and respond to any filtering, pagination and sorting actions performed by the user.

The proposed behaviour is as follows:

1. Pinned hits are always displayed above organic hits, not intermingled. Think oil on water.
2. Like organic hits, pinned results can be:
   a) subject to any structured filter selections (price, distance etc)
   b) accounted for in aggregations (typically shown in facet filters)
   c) pageable (meaning they disappear on page 2)
   d) not duplicated (a pinned hit should not also appear as an organic hit)
   e) highlighted
3. Unlike organic hits, they:
   a) are NOT filtered by free-text search input
   b) have an admin-supplied ranking order
   c) are NOT pinned when the primary sort order is anything other than relevance.

Pinned hits will not be offered as part of searches using field-collapsing - a flat list of pinned docs does not blend naturally when sat above search results that are organised under grouping keys.

The top_hits aggregation is another elasticsearch feature but is not commonly used in search UIs like the one shown above so it remains unclear what if any rules apply to pinning here.

Proposed solution - score manipulation

Given the above requirements it seems natural to think of this problem as finding a way to give a reliable boost to scores of selected documents to always place them above all other organic results. This can be achieved by wrapping the organic Query clauses in a new PinnedQuery which lists the results to be pinned e.g.:

GET test
{
  "query": {
	"pinned_query": {
	  "pinned_ids": [
		"9nht3GgBe1Qo_SrLzRCJ",
		"-Hht3GgBe1Qo_SrLzRCJ"
	  ],
	  "organic": {
		"match": {
		  "my_field": "my free-text query"
		}
	  }
	}
  }
}

In this example the pinned_ids contains the ids of documents to be placed at the top of the results, in order of relevance.
The organic clause contains any other valid elasticsearch query.
The pinned_query clause will essentially run a form of Boolean query where the IDs of the pinned docs are an "OR" to the clause in the organic clause.
The pinned_query would manipulate the floats of scores produced by the organic clause, reserving the greatest float exponent of scores to encode the positions of pinned results. This will ensure that pinned documents will be ranked in the right positions while preserving the ranking order of matches in the organic results.

Sometimes there will be additional criteria such as price-range filters that need to apply to both organic and pinned documents. These would be enforced by wrapping the pinned_query clause in a regular Boolean query alongside these additional constraints.

[elasticmachine]

Pinging @elastic/es-search