Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timestamp processor fails to parse date correctly #15012

Closed
fkelbert opened this issue Dec 9, 2019 · 10 comments
Closed

Timestamp processor fails to parse date correctly #15012

fkelbert opened this issue Dec 9, 2019 · 10 comments
Labels

Comments

@fkelbert
Copy link

fkelbert commented Dec 9, 2019

The Filebeat timestamp processor in version 7.5.0 fails to parse dates correctly. Only the third of the three dates is parsed correctly (though even for this one, milliseconds are wrong).

Input file:

13.06.19 15:04:05:001
03.12.19 17:47:38:402
03.04.19 15:04:05:999

filebeat.yml:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/test.txt
processors:
  - timestamp:
      field: message
      layouts:
        - '02.01.06 15:04:05:999'
output.console.pretty: true

Output:

{
  "@timestamp": "2019-12-09T15:28:55.973Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.0"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "agent": {
    "type": "filebeat",
    "ephemeral_id": "7cbeaeaf-4378-4703-8509-e43399081c16",
    "hostname": "fk",
    "id": "6525664c-da2b-4b49-a676-87e1e71d5591",
    "version": "7.5.0"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/tmp/test.txt"
    }
  },
  "message": "13.06.19 15:04:05:001",
  "input": {
    "type": "log"
  }
}
{
  "@timestamp": "2019-12-09T15:28:58.915Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.0"
  },
  "input": {
    "type": "log"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "agent": {
    "version": "7.5.0",
    "type": "filebeat",
    "ephemeral_id": "7cbeaeaf-4378-4703-8509-e43399081c16",
    "hostname": "fk",
    "id": "6525664c-da2b-4b49-a676-87e1e71d5591"
  },
  "log": {
    "offset": 22,
    "file": {
      "path": "/tmp/test.txt"
    }
  },
  "message": "03.12.19 17:47:38:402"
}
{
  "@timestamp": "2019-04-03T15:04:05.000Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.0"
  },
  "log": {
    "file": {
      "path": "/tmp/test.txt"
    },
    "offset": 44
  },
  "message": "03.04.19 15:04:05:999",
  "input": {
    "type": "log"
  },
  "agent": {
    "type": "filebeat",
    "ephemeral_id": "7cbeaeaf-4378-4703-8509-e43399081c16",
    "hostname": "fk",
    "id": "6525664c-da2b-4b49-a676-87e1e71d5591",
    "version": "7.5.0"
  },
  "ecs": {
    "version": "1.1.0"
  }
}
@sayden
Copy link
Contributor

sayden commented Jan 6, 2020

TLDR: Go doesn't accept anything apart of a dot . to parse milliseconds in date/time.

I have been doing some research and, unfortunately, this is a known issue in the format parser of Go language. golang/go#6189 In this issue they talk about commas but the situation is the same regarding colon. It's very inconvenient for this use case but all in all 17:47:38:402 (triple colon) is not any kind of known timestamp.

I couldn't find any easy workaround. Maybe some processor before this one to convert the last colon into a dot . (with the appropiate layout change, of course)

@fkelbert
Copy link
Author

fkelbert commented Jan 6, 2020

Thank you for doing that research @sayden.

As a user of this functionality, I would have assumed that the separators do not really matter and that I can essentially use any separator as long as they match up in my timestamps and within the layout description. I was thinking of the layout as just a "stencil" for the timestamp.

At the very least, such restrictions should be described in the documentation. Ideally, we would even provide a list of supported formats (if this list is of a reasonable lenvth). Users shouldn't have to go through https://godoc.org/time#pkg-constants

@CamilYed
Copy link

This still not working cannot parse?
`timestamp:
field: '@timestamp'
layouts:
- '2020-05-14T07:15:16.729Z'
test:
- '2020-05-14T07:15:16.729Z'

@sonalkr132
Copy link

TLDR: Go doesn't accept anything apart of a dot . to parse milliseconds in date/time.

Only true if you haven't displeased the timestamp format gods with a "non-standard" format. '2020-10-28 00:54:11.558000' is an invalid timestamp. 🙅‍♂️

 Exiting: error initializing processors: failed to parse test timestamp: failed parsing time field custom_time='2020-10-28 00:54:11.558000'

@matsgoran
Copy link

This rfc3339 timestamp doesn't seem to work either: '2020-12-15T08:44:39.263105Z'

Error while initializing input: failed to parse test timestamp: failed parsing time field vault.@timestamp='2020-12-15T08:44:39.263105Z'

@skldfm
Copy link

skldfm commented Jan 28, 2021

@AlekSi
Copy link

AlekSi commented Sep 3, 2021

JFYI, the linked Go issue is now resolved.

@botelastic
Copy link

botelastic bot commented Sep 3, 2022

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

@botelastic botelastic bot added Stalled needs_team Indicates that the issue/PR needs a Team:* label labels Sep 3, 2022
@mtojek mtojek added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Sep 5, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed needs_team Indicates that the issue/PR needs a Team:* label Stalled labels Sep 5, 2022
@botelastic
Copy link

botelastic bot commented Sep 5, 2023

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

@botelastic botelastic bot added the Stalled label Sep 5, 2023
@botelastic botelastic bot closed this as completed Mar 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

9 participants