Filebeat 6.0.0-rc1: json.overwrite not working with @timestamp

[friesoft]

• Version: 6.0.0-rc1
• Operating System: Red Hat 7
• Steps to Reproduce:

Forwarding json messages from a log file to logstash using Filebeat does not set the @timestamp attribute correcty. The timestamps are already off on the Filebeat side (console output).
With 5.0.1 and 5.6.3 (latest GA) it is working, with 6.0.0-rc1 we get duplicate @timestamp fields.

Sample JSON to reproduce the issue:
{ "@timestamp": "2017-10-17T10:03:14.301Z", "request": "/" }

Sample Filebeat Config 5.0.1/5.6.3 (working):

filebeat.prospectors:
- input_type: log
  paths:
    - serverlogs/apache.json
  json.keys_under_root: true
  json.add_error_key: true
  json.overwrite_keys: true
  fields_under_root: true

output.console:
  pretty: true

Sample Filebeat Config 6.0.0-rc1 (not working):

filebeat.prospectors:
- prospector_type: log
  paths:
    - serverlogs/apache.json
  json.keys_under_root: true
  json.add_error_key: true
  json.overwrite_keys: true
  fields_under_root: true

output.console:
  pretty: true

Running with a 5.0.1 installation (download, untar, add serverlogs/apache.json file with sample provided above, add filebeat.json.yml with sample provided above, run):

[friedreb@pc64901 filebeat-5.0.1-linux-x86_64]$ rm -rf data/ && ./filebeat -c filebeat.json.yml
{
  "@timestamp": "2017-10-17T10:03:14.301Z",
  "beat": {
    "hostname": "pc64901",
    "name": "pc64901",
    "version": "5.0.1"
  },
  "input_type": "log",
  "offset": 61,
  "request": "/",
  "source": "serverlogs/apache.json",
  "type": "log"
}

Running with a 6.0.0-rc1 installation (download, untar, add serverlogs/apache.json file with sample provided above, add filebeat.json.yml with sample provided above, run):

[friedreb@pc64901 filebeat-6.0.0-rc1-linux-x86_64]$ ./filebeat -c filebeat.json.yml
{
  "@timestamp": "2017-10-25T07:54:57.673Z",
  "@metadata": {
  "beat": "filebeat",
  "type": "doc",
  "version": "6.0.0-rc1"
  },
  "@timestamp": "2017-10-17T10:03:14.301Z",
  "beat": {
    "name": "pc64901",
    "hostname": "pc64901",
    "version": "6.0.0-rc1"
  },
  "source": "/products/filebeat-6.0.0-rc1-linux-x86_64/serverlogs/apache.json",
  "offset": 61,
  "request": "/"
}

You can clearly see that the 6.0.0-rc1 installation contains duplicate @timestamp fields. This is resulting in wrong timestamps.

In the discuss entry on elastic.co we (@kvch) agreed that this is a bug.
https://discuss.elastic.co/t/6-0-0-rc1-json-overwrite-keys-not-working-with-timestamp/105189

[tsg]

This was fixed in #5436, with some followups in #5437. Many thanks for reporting this issue!