Use UTC time when formatting timestamps (#3744)

[fjgal]

Solves issue #3744

Now the when the timestamp is stored in the context passed to the formatter is to converted to UTC. Some unit test (formatevents_test.go and datetime_test.go) had to be adapted to this behavior.

Modified GetIndex unit tests (elasticsearch/client_test.go) to expose this bug, now they use a timezone and a time that falls in between localtime and UTC to make sure the date in the index name is based on UTC.

[karmi]

Hi @fjgal, we have found your signature in our records, but it seems like you have signed with a different e-mail than the one used in yout Git commit. Can you please add both of these e-mails into your Github profile (they can be hidden), so we can match your e-mails to your Github profile?

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[fjgal]

Hi @karmi I have ammended the commit using the email address I have in my github account. Hope this is good now. Thanks for pointing this out!

[ruflin]

jenkins, test it

[ruflin]

@fjgal Thanks for taking a stab at this. CLA works now.

I need to have a closer look at the PR and all the parts this could affect to better understand if that is our way forward.

[ruflin]

@urso @tsg Could you comment here? I remember at least one of you was thinking more about timezones etc.

[urso]

As we store timestamps in UTC-0, this should be reflected in the index name creation as well..

I'm a little on the fence, though. This change potentially affects every use of format strings, forcing UTC-0 on every user.

The timestamp is inserted using the time format: %{+yyyy.MM.dd}. Alternatively we could modify/enhance the timestamp formatter to support UTC conversion internally. e.g. add timezone support to our data time formatter and change the default pattern to: %{+yyyy.MM.ddZ0}.

[ruflin]

We should probably have a broader discussion on where we expect to use UTC and where we allow the user to have time zone specific time stamps.

In general I would expect that the events which happen at the same time whereever they are in the world end up in the same index. At the same time I think an event must be able to indicate from which time zone it was sent and converted. I wonder if beat.timezone is enough here?

[jsoriano]

I think this is probably solved by the use of ILM in recent versions, when index timestamp doesn't depend on the content of the event.

Please reopen if this is causing issues in recent versions.