Fields API can lose precision when fetching dates

[nik9000]

We allow folks to specify dates like:

"d": 6123.123

Which you should read as 6123123000 nanoseconds since epoch. There are a lot of layers, but when we see this format we pretty much shift the decimal point six places to the right and parse the whole thing as nanoseconds. Then we use the standard java time utilities to convert that nanosecond precision instant into whatever the date's native format is. This all works fine on parsing.

But in a few places we let jackson deserialize the _source without type hints. When it sees a number like 6123.123 it automatically converts it to a double precision floating point value. This will lose precision for big dates. With date its mostly ok because double's maximum precise value is 9,007,199,254,740,992 which is something like the year 287396. If I live so long I'll be glad to fix the bugs. I say mostly here because there are some issues with rounding so we might end up being off by a millisecond. Sadly, with date_nanos we lose precision on any date after 6am, April 15th, 1970 UTC.

So, you rightly ask, when do we let jackson deserialize the _source? Well, when scripts access _source via params._source. And when we perform an _update action. And in the fields API. And maybe other places I haven't found yet.

Note: This describes the state we'll be in if we merge #70040.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[nik9000]

I pulled that big number from the wikipedia page on double precision floating point numbers. It's a good read.

[nik9000]

OOOOOOH! Another place where just fetch the source and let jackson do whatever is in _source fetching. But only when you add pretty. If you leave it off you get what you originally sent us:

curl -XDELETE -uelastic:password -HContent-Type:application/json localhost:9200/test
curl -XPUT -uelastic:password -HContent-Type:application/json localhost:9200/test -d'{
  "mappings": {
    "properties": {
      "d": {
        "type": "date"
      }
    }
  }
}'

curl -XPOST -uelastic:password -HContent-Type:application/json 'localhost:9200/test/_doc?refresh&pretty' -d'{"d": 1615237777000.123456}'
curl -XPOST -uelastic:password -HContent-Type:application/json localhost:9200/test/_search?pretty -d'{
  "fields": [
    {"field": "d"}
  ]
}'
#         "_source" : {
#           "d" : 1.6152377770001235E12 <----- That ain't what I sent!
#         },
#         "fields" : {
#           "d" : [
#             "2021-03-08T21:09:37.000Z"
#           ]
#         }


curl -XPOST -uelastic:password -HContent-Type:application/json localhost:9200/test/_search -d'{
  "fields": [
    {"field": "d"}
  ]
}'
# "_source":{"d": 1615237777000.123456}      <---- Ok!

curl -XPOST -uelastic:password -HContent-Type:application/json localhost:9200/test/_search -d'{
  "fields": [
    {"field": "d"}
  ]
}' | jq .
#         "_source": {
#           "d": 1615237777000.1235         <---- JQ breaks it too!
#         },
#         "fields": {
#           "d": [
#             "2021-03-08T21:09:37.000Z"
#           ]
#         }

[jpountz]

Would this benefit from #46531?

[nik9000]

Probably

…

On Mon, Mar 29, 2021, 06:14 Adrien Grand ***@***.***> wrote: Would this benefit from #46531 <#46531>? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#70085 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUXIUE777L3LF32NX3CHLTGBOIJANCNFSM4YZPXALA> .

[nik9000]

I think it'd make it possible to use the two code paths for parsing successfully. But maybe we don't want to use the two code paths anyway and should migrate to a from-x-content reader rather than from-map. Maybe. Dunno!

…

On Mon, Mar 29, 2021, 07:23 Nikolas Everett ***@***.***> wrote: Probably On Mon, Mar 29, 2021, 06:14 Adrien Grand ***@***.***> wrote: > Would this benefit from #46531 > <#46531>? > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#70085 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AABUXIUE777L3LF32NX3CHLTGBOIJANCNFSM4YZPXALA> > . >

[nik9000]

A bunch of folks got together and thought about this for a while. We decided to do three things "real soon":

• Update the docs for date field to explain that they are based on strings and to advise against sending numbers with decimal points. (Advise against dates with decimal points #71578)
• Make sure the source filtering API doesn't lose precision on these numbers.
• Make sure the fields API doesn't lose precision on these dates.

To well and truly fix this issue would require a bunch of breaking changes that we think would be rough on folks. Or it'd require a bunch of advancements in scripts that aren't "soon" things.

[skumarp7]

When can we expect the below action items to be addressed?

1. Make sure the source filtering API doesn't lose precision on these numbers.
2. Make sure the fields API doesn't lose precision on these dates.

[javanna]

This issue stalled over time, I believe due to the breaking nature of some of the needed changes. We have been discussing though adding support for loading from stored fields as well as doc_values , which is not a direct fix for the precision loss, but it could help if we avoided loading from _source in this specific case. Sounds like a hack, but I thought I would bring it up to see what others think.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)