Fix issue with returning incomplete fragment for plain highlighter.

[ioanatia]

This is a bug that was noticed in the context of App Search which uses the plain highlighter.
It seems that when the text field value is ending in special characters and no_match_size is equal or greater than the size of the text, we are not returning the full text.
This is because in findGoodEndForNoHighlightExcerpt we are consuming the token stream and we just return the position of where the last token ends, thus ignoring any special characters the original text might end with that are not part of a token.

Steps to replicate:

POST my-index/_doc/1?refresh=true
{
  "my_text_field": ")hey)"
}

GET my-index/_search
{
  "query": {
    "bool": {
      "must": {
        "match_all": {}
      }
    }
  },
  "highlight": {
    "type": "plain",
    "fields": {
      "my_text_field": {
        "fragment_size": 100,
        "no_match_size": 100
      }
    },
    "highlight_query": {
      "match_all": {}
    }
  }
}

response - the highlight is )hey instead of )hey):

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "my-index",
        "_id": "1",
        "_score": 1,
        "_source": {
          "my_text_field": ")hey)"
        },
        "highlight": {
          "my_text_field": [
            ")hey"
          ]
        }
      }
    ]
  }
}

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Hi @ioanatia, I've created a changelog YAML for you.

[mayya-sharipova]

Should we move this to like 216, even before analyzing contents string?

[ioanatia]

Was asking myself the same, but I think the behaviour would not be consistent which could cause confusion - when no offsets are provided we would some return highlights for some document fields, but not for others and the reason why that happens might not be apparent.
If that's not actually a problem I am okay to change it.

[mayya-sharipova]

Thinking more about it, you are right. It is better to have the consistent behaviour. Let's keep your changes as they are.

[mayya-sharipova]

@ioanatia Thanks for fixing, +1 to merge after addressing my small. comment

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
✅	8.15
❌	7.17	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 110707