Formatted sort values for search_after

[jimczi]

Today the sort values used to rank each hit in the response are exposed as raw values in an array (response.hits.hit.0.sort).
These values are meant to be copied in search_after request in order to paginate efficiently over a set of results.

By default, the sort value for date and date_nanos field is represented as a long, that's the internal representation that we use for this field. This leaking of internal representation is problematic because the returned value cannot be interpreted without context. date returns the number of milliseconds since epoch while date_nanos returns the number of nanoseconds. In order to fix this discrepancy we'd like to gradually introduce formatted sort values.

At first we'd like to add a format option to any sort value in a search request. Setting a format there would ensure that the sort values in the response would be formatted accordingly:

{
    "query": {
        "match_all": {}
    },
    "sort": [
        {
            "field": "timestamp",
            "format": "strict_date_optional_time_nanos"
        }
    ]
}

The same format would also be used to parse the search_after value so that copying the sort values directly in search_after continues to work:

{
    "query": {
        "match_all": {}
    },
    "sort": [
        {
            "field": "timestamp",
            "format": "strict_date_optional_time_nanos"
        }
    ],
    "search_after": [
        "2015-01-01T12:10:30.123456789Z"
    ]
}

It would be nice to also apply the formatter of the field by default if no format is specified. That would solve the leaking of internal representation entirely but would have more impact on users.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[mayya-sharipova]

@jimczi Thanks for the proposal, Jim.

This proposal is also relevant and useful for unsigned_long which returns sort values of long or BigInteger type depending on a value, and there was a request to return unsigned_long values are String.

I am also wondering about having format for other parts of a search request, e.g. field, docvalues_fields etc. May be in this case, it makes sense to have format as a separate section of a search request so that it can be applied in every part (sort, fields ...), e.g:

{
  "query": {
    "match_all": {}
  },
  "formats": [
    {
      "field": "timestamp",
      "format": "strict_date_optional_time_nanos"
    },
    {
      "field": "my_unsigned_long",
      "format": "string"
    }
  ],
  "sort": ["timestamp", "my_unsigned_long"],
  "fields": ["timestamp", "my_unsigned_long"]
}

[dnhatn]

@mayya-sharipova While your proposal avoids specifying the format multiple times, I think the implementation is a bit more complicated and BWC is also an issue. Also, if a user wants the highest resolution for sort, but a lower resolution for fields, then it's not possible with a single format session.

I've open #70357 for this. It would be great if you can take a look.