Highlighters shouldn't error on big documents

[markharwood]

For the casual searcher it is not particularly helpful to have elasticsearch return an error if they are unlucky enough to match a big document.
The user gets a 400 error with this sort of message:

The length of [xxx] field ... has exceeded [1000000] - maximum allowed to be analyzed 
for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] 
index level setting. For large texts, indexing with offsets or term vectors is recommended!

At this point the only workarounds the user has are:

a) User rewrites query with a NOT clause to exclude IDs of rogue docs (not ideal)
b) User reindexes content with offsets (a pain)
c) User reindexes content and truncates long strings e.g. with an ingest processor (not ideal)
d) User increases the max_analyzed_offset setting (not ideal)

None of these are great options so the proposal is that highlighters could be prevented from throwing an error and instead use a cheaper "fallback" approach to highlighting e.g. returning the first N characters of a large string field. The open questions are:

1. Do we need additional properties in the highlight request to define the fallback approach?
2. How do we warn the user that a fallback approach was applied for a particular result?
3. Will some users want the old behaviour of errors rather than fallbacks?

[elasticmachine]

Pinging @elastic/es-search (:Search/Highlighting)

[mayya-sharipova]

I am also wondering what if the first N characters don't contain any query terms to highlight? Is it ok to present a user with empty highlights in this case?

[markharwood]

Is it ok to present a user with empty highlights in this case?

For the Kibana use case I imagine having the Discover page showing some content (maybe with a ... to indicate there's more) would be better than a blank space or an error.

[markharwood]

We discussed this today and had a proposal for how users can avoid this error without resorting to any of the painful workarounds listed above.
As part of the highlight request a user could provide a form of query-time `"ignore_above" statement:

"ignore_above" : {
     "size": 10000,
     "replacement_value": "Too large ..."
}

The size parameter specifies the maximum field size (where field size is the length in bytes of a value or, if an array, the sum of all element value lengths).
The replacement_value is a bit like the null value setting in index mappings - it's a user-provided string that they can recognise as indicating failing content.
The proposal was that this ignore_above option could be used for all highlighters and tweaked according to the user's taste. Obviously it wouldn't be useful to set size to a value greater than the index.highlight.max_analyzed_offset setting because this would potentially trigger an error rather than the desired ignore behaviour.

[eedugon]

@markharwood : Apparently this error is also raised with fields that are not even indexed.

Taking a look at this mapping:

"base64Message": {
      "type": "text",
      "index": false
     }

It also generates errors as Kibana uses highlighting by default:

"The length of [doc.base64Message] field of [AXBikvAZbAZqpEylLWf0] doc of [iib-busnl-prd-service-base64messages-2020.02] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!"

So, we have keywords with ignore_above setting, which is ignored by the highlighter (#43800) and where the offsets workaround is not possible, and we have also fields with index: false option where we cannot apply either the offsets workaround.

Just for your consideration for improving the highlighter. Don't really know if the fix should me more at Kibana side when using the highlighter (selecting better options) or at the highlighter engine to avoid failing search requests due to this.

[mayya-sharipova]

@eedugon Thanks for letting us know, we will try to address your use case as well

[markharwood]

We discussed this today and came up with two additional proposals:

1. More pre-empting: like a document whose value has exceeded the ignore_above setting for a keyword field, we should avoid even trying to highlight fields that are known to be unindexed.
2. Better fallbacks - rather than only flagging a field was too large in the response we should also consider providing a best-efforts option of highlighting only the first N characters of a document.