Skip to content

Commit

Permalink
Add ERR to ranking evaluation documentation
Browse files Browse the repository at this point in the history
This change adds a section about the Expected Reciprocal Rank metric (ERR) to
the Ranking Evaluation documentation.
  • Loading branch information
Christoph Büscher committed Jul 24, 2018
1 parent d07b4ec commit 48121d8
Showing 1 changed file with 50 additions and 0 deletions.
50 changes: 50 additions & 0 deletions docs/reference/search/rank-eval.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,56 @@ in the query. Defaults to 10.
|`normalize` | If set to `true`, this metric will calculate the https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG[Normalized DCG].
|=======================================================================

[float]
==== Expected Reciprocal Rank (ERR)

Expected Reciprocal Rank (ERR) is an extension of the classical reciprocal rank for the graded relevance case
(Chapelle, Olivier, Donald Metzler, Ya Zhang, and Pierre Grinspan. 2009.
http://olivier.chapelle.cc/pub/err.pdf[Expected reciprocal rank for graded relevance].)

It is based on the assumption of a cascade model of search, which models that a user scans through ranked search
results in order and stops at the first document satisfies the information need of the user. For this reason, it
is a good metric for question answering and navigation queries, but less for survey oriented information needs
where the user is interested in finding several relevant documents in the top k results.

The metric tries to model the expectation of the reciprocal of the position of a result at which a user stops.
This means, relevant document in top ranking positions will contribute much to the overall ERR score. The same
document will contribute much less to the score on a lower rank, but even more so if there were some
relevant documents preceding it. By this, ERR discounts documents which are shown below very relevant documents
and introduces some kind of dependency in the ordering of relevant documents.

[source,js]
--------------------------------
GET /twitter/_rank_eval
{
"requests": [
{
"id": "JFK query",
"request": { "query": { "match_all": {}}},
"ratings": []
}],
"metric": {
"expected_reciprocal_rank": {
"maximum_relevance" : 3,
"k" : 20
}
}
}
--------------------------------
// CONSOLE
// TEST[setup:twitter]

The `expected_reciprocal_rank` metric takes the following parameters:

[cols="<,<",options="header",]
|=======================================================================
|Parameter |Description
| `maximum_relevance` | Mandatory parameter. The highest relevance grade used in the user supplied
relevance judgments.
|`k` | sets the maximum number of documents retrieved per query. This value will act in place of the usual `size` parameter
in the query. Defaults to 10.
|=======================================================================

[float]
=== Response format

Expand Down

0 comments on commit 48121d8

Please sign in to comment.