Filebeat 6.0.0-rc1: logfiles are fully parsed on every start

[friesoft]

• Version: 6.0.0-rc1, 5.6.3 (latest GA)
• Operating System: Red Hat 7
• Steps to Reproduce:
  Restarting Filebeat re-reads and parses the same log lines (json) every time. Can be seen easily with the console output of Filebeat. On first run it correctly outputs the parsed logline, on second, thirds, .... run it outputs it again.

Sample JSON to reproduce the issue:
{ "@timestamp": "2017-10-17T10:03:14.301Z", "request": "/" }

Sample Filebeat Config 5.0.1 (working) and 5.6.3 (not working):

filebeat.prospectors:
- input_type: log
  paths:
    - serverlogs/apache.json
  json.keys_under_root: true
  json.add_error_key: true
  json.overwrite_keys: true
  fields_under_root: true

output.console:
  pretty: true

Sample Filebeat Config 6.0.0-rc1 (not working):

filebeat.prospectors:
- prospector_type: log
  paths:
    - serverlogs/apache.json
  json.keys_under_root: true
  json.add_error_key: true
  json.overwrite_keys: true
  fields_under_root: true

output.console:
  pretty: true

Running with a 5.0.1 installation (download, untar, add serverlogs/apache.json file with sample provided above, add filebeat.json.yml with sample provided above, run):

[friedreb@pc64901 filebeat-5.0.1-linux-x86_64]$ ./filebeat -c filebeat.json.yml
{
  "@timestamp": "2017-10-17T10:03:14.301Z",
  "beat": {
    "hostname": "pc64901",
    "name": "pc64901",
    "version": "5.0.1"
  },
  "input_type": "log",
  "offset": 61,
  "request": "/",
  "source": "serverlogs/apache.json",
  "type": "log"
}

Exiting filebeat (Ctrl+C) and starting it again doesn't produce any output on 5.0.1 as it isn't parsing anything as there are no new loglines (the one line provided as a sample has been parsed already during last run).

Running with a 5.6.3 or 6.0.0-rc1 installation it produces the same loglines on every start (download, untar, add serverlogs/apache.json file with sample provided above, add filebeat.json.yml with sample provided above, run):

[friedreb@pc64901 filebeat-6.0.0-rc1-linux-x86_64]$ ./filebeat -c filebeat.json.yml
{
  "@timestamp": "2017-10-25T07:54:57.673Z",
  "@metadata": {
  "beat": "filebeat",
  "type": "doc",
  "version": "6.0.0-rc1"
  },
  "@timestamp": "2017-10-17T10:03:14.301Z",
  "beat": {
    "name": "pc64901",
    "hostname": "pc64901",
    "version": "6.0.0-rc1"
  },
  "source": "/products/filebeat-6.0.0-rc1-linux-x86_64/serverlogs/apache.json",
  "offset": 61,
  "request": "/"
}

Every time I exit filebeat and start it again the same lines are parsed and output (of course every time with a new parse timestamp (the duplicate timestamps is issue #5440)

In the discuss entry on elastic.co we (@kvch) agreed that this is a bug.
https://discuss.elastic.co/t/6-0-0-rc1-json-overwrite-keys-not-working-with-timestamp/105189

[tsg]

The issue seems to happen when relative paths are used in the paths configuration. A workaround is to use absolute paths. I'm working on a fix and will open a PR shortly.