Range query on date field applies date math/rounding when not requested

[mvdz]

Elasticsearch version (bin/elasticsearch --version):
Version: 7.9.1, Build: default/tar/083627f112ba94dffc1232e8b42b73492789ef91/2020-09-01T21:22:21.964974Z, JVM: 14.0.1

Plugins installed: []

JVM version (java -version):
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.1+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7, mixed mode, sharing)

OS version (uname -a if on a Unix-like system):
Linux 4.15.0-109-generic #110-Ubuntu SMP Tue Jun 23 02:39:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
(Ubuntu 18.04)

Description of the problem including expected versus actual behavior:
A document with a date/time value "2020-01-01T12:00:00.500Z" is not returned by the range query "gt": "2020-01-01T12:00:00Z".

A range query on a date field supports date math and rounding, according to:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html#ranges-on-dates
This documents that rounding can be requested by appending for example "||/s". But the behaviour here is rounding even when given a date/time without the date maths syntax.

This caused us problems because when using java.time for creating these queries the default ISO_INSTANT formatter is specified as "The nano-of-second outputs zero, three, six or nine digits as necessary.", thus not providing milliseconds when these happen to be 0.

A workaround is to use a different formatter which always outputs the milliseconds, or explicitly using date maths without any actual maths/rounding (by appending "||").

Steps to reproduce:

curl -s -XDELETE localhost:9200/date-test?pretty

# create index with date field mapping
curl -s -XPUT localhost:9200/date-test?pretty -H'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "datetime": {
        "type": "date"
      }
    }
  }
}'

# index 2 documents
curl -s -XPOST localhost:9200/date-test/_doc/?pretty\&refresh=true -H'Content-Type: application/json' -d'
{
  "datetime": "2020-01-01T12:00:00.500Z"
}'

curl -s -XPOST localhost:9200/date-test/_doc/?pretty\&refresh=true -H'Content-Type: application/json' -d'
{
  "datetime": "2020-01-01T12:00:01.500Z"
}'

# search with a date range
curl -s localhost:9200/date-test/_search?pretty -H'Content-Type: application/json' -d'
{
  "query": {
    "range": {
      "datetime": {
        "gt": "2020-01-01T12:00:00Z"
      }
    }
  }
}'

Expected result: both documents should be returned.
Actual result: only the second document is returned.

Changing the query to "gt": "2020-01-01T12:00:00.000Z" does behave as expected, returning both documents.

Changing the query to "gt": "2020-01-01T12:00:00Z||" also returns both documents.

Changing the query to "gt": "2020-01-01T12:00:00.000Z||/s", explicitly enabling rounding only returns the second document as expected.

Provide logs (if relevant):
Not relevant

[pgomulka]

all you mentioned is right.
with range query and not providing nano second part we will try to round it to 999_999_999.

I think you can either change your formatter (ISO_INSTANT is a formatter from your app?), preprocess your data with ingest pipeline or script or just use

"gt": "2020-01-01T12:00:00Z||"

this I think is undocumented, but you are effectively asking to round on 'none' field so getting a date without rounding.

[mvdz]

Ah with the ISO_INSTANT I meant to refer to the Java Time library java.time.format.DateTimeFormatter#ISO_INSTANT, which is also the formatter by java.time.Instant#toString. This is the formatter we were using since it's the default in Java.

We now switched to a custom formatter which always includes milliseconds even if they are 0, to work around this rounding behaviour.

I'm not sure if you're saying that this rounding is behaving as intended? It was certainly surprising for us, especially since the mentions I could find in the documentation about rounding dates is that you can do it explicitly with the "||/s" syntax, not that it is also done implicitly. If this is really intended then I think a warning in the range filter documentation (and elsewhere it may apply) would be appropriate.

[pgomulka]

yes, the behaviour is correct. I will try to improve the doc around rounding when using date field in range queries

[pgomulka]

closing as behaviour is correct - improved the docs #63109