-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add processor to set timezone for an event #3902
Conversation
Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run. |
1 similar comment
Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run. |
|
||
func newAddLocale(c common.Config) (processors.Processor, error) { | ||
config := struct { | ||
TimeZone string `config:"timezone"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this just be completely automatic using time.Now().Location()
? Example: https://play.golang.org/p/2Lpzo3KvWJ
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's what i not understand correctly. I thought we give the user the option to set the timezone. If not so you are right.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think idea is to report the local timezone. It probably shouldn't be configurable at all. Just detect the local TZ and report it in the event.
Once you have the timezone, you can use Logstash to do further transformations on the data (like interpreting syslog dates that are missing the timezone) or converting @timestamp
to the machine's local timezone (not recommended IMHO).
A more specific use case is analyzing logon events, the timezone can be used to determine if the event occurred during normal working hours. Then you can alert on people working after hours and send them on a vacation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like there's an even more direct way: time.Local.String()
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
time.Local.String()
only gives you Local
. I have changed the method to get the the timezone
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, am I misunderstanding the docs at https://golang.org/pkg/time/#Location ?
Local represents the system's local time zone.
var Local *Location = &localLoc
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On golang Playground i get UTC
. On my machine i get
"beat": {
"hostname": "4201halwsd00001",
"name": "4201halwsd00001",
"timezone": "Local",
"version": "6.0.0-alpha1"
},
???
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, you are right. I get the same thing "Local". So zone, _ := time.Now().Zone()
should be sufficient because Now()
always returns local time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok. Changed up to time.Now().Zone()
. I was not sure if Now()
always returns local time.
} | ||
|
||
func (l addLocale) Run(event common.MapStr) (common.MapStr, error) { | ||
zone, err := time.LoadLocation(l.timezone) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Run
method should be as simple and fast as possible. In newAddLocale()
you I think can do this work once and store the result so that Run just becomes a map.Put()
operation.
} | ||
|
||
func newAddLocale(c common.Config) (processors.Processor, error) { | ||
config := struct { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This whole config definition and the Unpack
call can now be removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
Can we use this for the changelog? |
@maddin2016 Changelog entry LGTM. Don't forget to add the PR number at the end (see other lines) |
The config.full.yml files should mention the new processor. https://github.com/elastic/beats/blob/master/libbeat/_meta/config.full.yml#L45-L46 |
8d3f1b3
to
b214b2d
Compare
b214b2d
to
a06dd06
Compare
Hi, |
Hi @ohadravid. For now we only export the timzone as a name |
@ohadravid sorry 🙈 . You are right. Now we have |
Servers traveling in time ... 🙈 We should report the time zone without DST. BTW this reminds me we will need to add docs to show people what the output of this will be and how to configure it. |
I think it will be more generic to just re-read the value at |
Tested locally and timzone is exported as |
@maddin2016 consider these two cases:
To me it is very important we re-read the timezone value on every event, so we can provide reliable data about the event's time as much as we can (For example, we might want to search for logins at odd hours (middle of the night) on the laptop). Also, it's really nice to be able to see and be a part of the development effort. Thank you for making this so transparent :) |
@ohadravid Thanks for the inputs. Laptops moving from one time zone to an other is definitively something I was not thinking of. Just changed my opinion about the above then :-) @andrewkroh WDYT? In case the processor is enabled, that means it is "refreshed" every time an even is created which can be up to 100k/s. In case we hit some issue here in the future we can think about caching it at least for 1s or something similar, but lets hit that wall first. |
- add_locale: | ||
------------------------------------------------------------------------------- | ||
|
||
NOTE: Please consider that `add_locale` differentiate between DST and regular time. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/differentiate/differentiates/
The |
With changes due to daylight savings and laptops moving around the world, making the timezone automatically update is a requirement. So let's do that. But I would like to understand the impact. @maddin2016 could you run two benchmarks, one as the code is now, and one where it reads the timezone on each |
@maddin2016 Thanks for the benchmarks. I would suggest to keep the benchmark code in the tests as it will be also useful to have these in the future. BTW: Don't forget |
What about to move the logic into the run method? |
@maddin2016 Which logic are you referring to? |
Sorry. I meant the read for timezone. |
} | ||
|
||
func (l addLocale) Run(event common.MapStr) (common.MapStr, error) { | ||
event.Put("beat.timezone", l.timezone) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given the requirement to have this field automatically reflect the latest timezone, let's not cache the timezone
like I requested previous (sorry for the change) and make Run
always call time.Now().Zone()
. Based on the benchmark results there wasn't a huge cost to doing this.
40efa97
to
f5f4b4c
Compare
This PR add a new processor to set timzone for an event. See #3867