Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Index pattern always treats field named "_timestamp" as a date regardless of mapping #121710

Closed
danielharada opened this issue Dec 20, 2021 · 3 comments · Fixed by #121772
Closed
Labels
bug Fixes for quality problems that affect the customer experience Feature:Data Views Data Views code and UI - index patterns before 8.0 impact:needs-assessment Product and/or Engineering needs to evaluate the impact of the change. loe:small Small Level of Effort

Comments

@danielharada
Copy link

Kibana version:
7.16.1

Elasticsearch version:
7.16.1

Server OS version:
ESS

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.):

Describe the bug:
When creating an index pattern, if you have a field named _timestamp it will always be listed as an option for the time field for the index pattern, even if this field is already mapped as a non date type. Treating this field as a date causes issues in Discover, which fails to load index patterns that have this field due to date formatting being applied to a non-date data type, such as :

Cannot retrieve search results

error fetching [_timestamp]: Field [_timestamp] of type [float] doesn't support formats.

Steps to reproduce:

  1. Create a document that has a _timestamp field:
PUT /timestamp_test/_doc
{
  "message": "doc 1",
  "@timestamp": "2021-12-20T18:27:23.436Z",
  "_timestamp": 1391759952.705
}
  1. Verify that your _timestamp field has a numeric mapping with GET /timestamp_test/_mapping
  2. Create an index pattern that captures the timestamp_test index, set @timestamp as the time field. Note that _timestamp was an option for the time field
  3. Check the newly created index pattern, verify that _timestamp is correctly shown as a numeric field type
  4. View the data in this index pattern in Discover
  5. Get error that _timestamp doesn't support formats

Expected behavior:
If _timestamp isn't mapped as a date, we shouldn't try to to date things to it.

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

@danielharada danielharada added the bug Fixes for quality problems that affect the customer experience label Dec 20, 2021
@botelastic botelastic bot added the needs-team Issues missing a team label label Dec 20, 2021
@rashmivkulkarni rashmivkulkarni added Feature:Discover Discover Application and removed needs-team Issues missing a team label labels Dec 20, 2021
@botelastic botelastic bot added the needs-team Issues missing a team label label Dec 20, 2021
@rashmivkulkarni rashmivkulkarni removed the needs-team Issues missing a team label label Dec 20, 2021
@botelastic botelastic bot added the needs-team Issues missing a team label label Dec 20, 2021
@rashmivkulkarni rashmivkulkarni added the Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. label Dec 20, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Dec 20, 2021
@tsullivan
Copy link
Member

I can reproduce this.

After creating the index pattern, the list of fields has the correct type for my _timestamp field:
image
But when running the search, the field is added to the query as a date_time field:
image
@elastic/kibana-app-services any idea why this { "field": "_timestamp", "format": "date_time" }, is in the request, instead of "format": "float" ? Is something hardcoded in the search API?

@tsullivan
Copy link
Member

When looking at requests fired before the search request, there is a request for this endpoint:
<host>/api/index_patterns/_fields_for_wildcard?pattern=sparse*&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score.

The response contains this:

{
  "fields": [
    ....
    {
      "name": "_timestamp",
      "type": "date",
      "esTypes": [
        "float"
      ],
      "searchable": true,
      "aggregatable": true,
      "readFromDocValues": true,
      "metadata_field": false
    },
    ....
  ]
}

@mattkime mattkime added Feature:Data Views Data Views code and UI - index patterns before 8.0 Team:AppServicesSv and removed Feature:Discover Discover Application Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. labels Dec 21, 2021
@exalate-issue-sync exalate-issue-sync bot added impact:needs-assessment Product and/or Engineering needs to evaluate the impact of the change. loe:small Small Level of Effort labels Feb 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience Feature:Data Views Data Views code and UI - index patterns before 8.0 impact:needs-assessment Product and/or Engineering needs to evaluate the impact of the change. loe:small Small Level of Effort
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants