Bad date range filter when filtering non-timefield date fields

[feicipet]

Hi,

I'm using Kibana version 4.4.1 with ES 2.2.0 from the Debian repos.

I have a field with the following type defined:

            "InvitationTime" : {
                "type" : "date",
                "format" : "dd/MM/yyyy HH:mm:ss Z"
            }

I created a data table visualization with a date histogram aggregation on this field:

When I click on one of the dates to filter though, I get an error where it tries to parse the millis-since-epoch value of the field in my field format:

Is this a bug or am I doing something wrong?

Thanks
Wong

[rashidkpc]

Legit bug

It looks like we aren't passing the format when creating range filters that use a date field. Now that Elasticsearch no longer accepts epoch by default we'll need to check if the field is a date, and if it is we need to tell it we're supplying epoch millis

This happens when creating a date_histogram agg on date field which is not the timefield for the index pattern.

Here's how we fix this:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html#_date_format_in_range_queries

[feicipet]

Thanks Rashid.

FWIW, I've found a workaround by specifying the format of the date field as follows:

        "ProcessDatetime" : {
            "type" : "date",
            "format" : "dd/MM/yyyy HH:mm:ss Z||epoch_millis"
        },

May be useful for anyone who's facing this issue while we wait for a fix.

Rashid, if you can explain why this "fixes" the problem, it'd be really educational for me :) I roughly surmised that doing this would allow ES to accept two formats of input but not sure if I'm right on that.

[ycombinator]

Rashid, if you can explain why this "fixes" the problem, it'd be really educational for me :) I roughly surmised that doing this would allow ES to accept two formats of input but not sure if I'm right on that.

@feicipet Hi, as described in https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html#multiple-date-formats, the || separator tells Elasticsearch to try various formats, one after the other, when parsing the string in the field as a timestamp. In your workaround, Elasticsearch first tries to parse the integer in Kibana's range filter request using the first format for your field, dd/MM/yyyy HH:mm:ss Z. When that doesn't work, it tries parsing it using the next format, epoch_millis, which succeeds.

As you noted this is a workaround; the real fix ought to be on Kibana's end as Rashid noted in his comment above.

[dimileeh]

This issue still doesn't seem to have been fixed in Kibana 4.5. As suggested by @feicipet, I had to add ||epoch_millis format for the date field in the index template to make Kibana apply the filters correctly when I click on a range in the histogram.

[ycombinator]

Hi @dlihhats, the issue has been fixed in Kibana 5. So you will need to upgrade to it to get the fix. Currently Kibana 5 is in an alpha release, which you can download from here: https://www.elastic.co/downloads/kibana (scroll down the page or search for "alpha").

[ddrake17]

This is broken again in the latest Kibana 6.1.2 I used the work around as suggested by @feicipet and it is working now.