Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data_nanos based search_after query with custom timestamp format causes exception #52424

Closed
kertal opened this issue Feb 17, 2020 · 1 comment · Fixed by #60328
Closed

data_nanos based search_after query with custom timestamp format causes exception #52424

kertal opened this issue Feb 17, 2020 · 1 comment · Fixed by #60328
Assignees
Labels
:Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team

Comments

@kertal
Copy link
Member

kertal commented Feb 17, 2020

Elasticsearch version (bin/elasticsearch --version):
Master

JVM version (java -version):

OS version (uname -a if on a Unix-like system):

Darwin Kernel Version 18.7.0

Description of the problem including expected versus actual behavior:

A query using search_after based on an index using a date_nanos with a custom timestamp format throws an exception

Steps to reproduce:

  1. Create custom mapping
PUT date_nanos_custom_timestamp
{
    "mappings": {
      "properties": {
        "timestamp": {
          "format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
          "type": "date_nanos"
        }
      }
    },
    "settings": {
      "index": {
        "number_of_replicas": "1",
        "number_of_shards": "1"
      }
    }
}
  1. Add same sample data
PUT date_nanos_custom_timestamp/_doc/1
{
  "timestamp": "2019-10-21 00:30:04.828740"
}

PUT date_nanos_custom_timestamp/_doc/2
{
  "timestamp": "2019-10-21 08:30:04.828733"
}
  1. Search
POST date_nanos_custom_timestamp/_search
{
  "size": 5,
  "search_after": [
    "1571617804828740000",
    1
  ],
  "sort": [
    {
      "timestamp": {
        "order": "asc",
        "unmapped_type": "boolean"
      }
    },
    {
      "_doc": {
        "order": "asc",
        "unmapped_type": "boolean"
      }
    }
  ]
  }
}

Now you're getting an exception:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "failed to parse date field [1571617804828740000] with format [yyyy-MM-dd HH:mm:ss.SSSSSS]: [failed to parse date field [1571617804828740000] with format [yyyy-MM-dd HH:mm:ss.SSSSSS]]"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "date_nanos_custom_timestamp",
        "node" : "059iRHeFT-ObiKskYJ_29w",
        "reason" : {
          "type" : "parse_exception",
          "reason" : "failed to parse date field [1571617804828740000] with format [yyyy-MM-dd HH:mm:ss.SSSSSS]: [failed to parse date field [1571617804828740000] with format [yyyy-MM-dd HH:mm:ss.SSSSSS]]",
          "caused_by" : {
            "type" : "illegal_argument_exception",
            "reason" : "failed to parse date field [1571617804828740000] with format [yyyy-MM-dd HH:mm:ss.SSSSSS]",
            "caused_by" : {
              "type" : "date_time_parse_exception",
              "reason" : "Text '1571617804828740000' could not be parsed at index 0"
            }
          }
        }
      }
    ]
  },
  "status" : 400
}

This worked a while ago, I've created a functional test in Kibana

elastic/kibana#54089

But had to skip it, since it caused the 400

elastic/kibana#56178

This works when providing the sort params as number

 "search_after": [
    1571617804828740000,
    1
  ],

But that's no a solution due to Browser limitation with BigInt numbers

@romseygeek romseygeek added the :Search/Search Search-related issues that do not fall into other categories label Feb 17, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (:Search/Search)

@rjernst rjernst added the Team:Search Meta label for search team label May 4, 2020
@nik9000 nik9000 self-assigned this Jun 12, 2020
nik9000 added a commit to nik9000/elasticsearch that referenced this issue Jul 28, 2020
This fixes `search_after` to properly parse string formatted dates that
have nanosecond resolution.

Closes elastic#52424
nik9000 added a commit that referenced this issue Jul 29, 2020
This fixes `search_after` to properly parse string formatted dates that
have nanosecond resolution.

Closes #52424
nik9000 added a commit to nik9000/elasticsearch that referenced this issue Jul 29, 2020
This fixes `search_after` to properly parse string formatted dates that
have nanosecond resolution.

Closes elastic#52424
nik9000 added a commit that referenced this issue Aug 3, 2020
…0426)

Allows nanosecond resolution in search_after (#60328)

This fixes `search_after` to properly parse string formatted dates that
have nanosecond resolution.

Closes #52424
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants