-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added Unix epoch timestamp support for JSON parser #4633
Conversation
…t and doc updated. If parsing a Unix epoch timestamp, e.g. 1536092344000, "json_time_format" config must be set to "Unix"; corresponding JSON value must be in milliseconds and unquoted.
Thanks for the pr, can you rename the special value "unix" or make it case insensitive to keep style with the rest of the config? It should handle only seconds, not ms, but could support a decimal value like |
Added "unix_ms" variation to handle epoch in milliseconds. Both values are now case insensitive. Docs and tests updated accordingly.
Hi @danielnelson, thanks for the feedback! I just commited the requested changes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's add support for string fields before we merge? More details in my comments.
… no decimal precision loss. For decimal unix timestamps coming as numeric JSON fields, there can be precision loss due to `float64` conversion at `JSONFlattener`. Docs and tests updated accordingly.
One last thing, now that we have quite a bit of code for parsing the timestamps I think we should extract it into a new function. |
Agreed. Just commited the change. Now there's a new function called |
I'm breaking my head over this. When I use the "unix" time_format, i get the following error; |
Hi @Thidjs, are you sure you're using a version with this change in it? If using a nightly build, it could be that it was packaged before the change was merged. |
@drenizg I tested again today with the latest nightly build and it seems to be working now. Thanx! |
closes: #4607
If parsing a Unix epoch timestamp in seconds, e.g. 1536092344.1,
json_time_format
config must be set to "unix" (case insensitive); corresponding JSON value can have a decimal part and can be a string or a number JSON representation.If value is in number representation, it'll be treated as a double precision float, and could have some precision loss. If value is in string representation, there'll be no precision loss up to nanosecond precision. Decimal positions beyond that will be dropped.
If parsing a Unix epoch timestamp in milliseconds, e.g. 1536092344100, this config must be set to "unix_ms" (case insensitive); corresponding JSON value must be a (long) integer, and can be in number or string JSON representation.
Required for all PRs: