Telegraf 1.5.3 and 1.6 Logparser quit logging after logfile is recreated/rotated

[zoom-scott-sancetta]

Bug report

Running Telegraf 1.5.3 or 1.6.0 on Redhat Linux, using the logparser plugin.
Every night certain logfiles are rotated via org.apache.log4j.DailyRollingFileAppender.
Our telegraf is watching these logs, and when they get renamed, telegraf stops collecting updates - maybe telegraf is holding on the the file handle… I have tried watch_method = poll AND inotify.
THIS WAS REPORTED AND APPARENTLY FIXED IN TELEGRAF 1.3, ISSUE #2967

Relevant telegraf.conf:

[[inputs.logparser]]
name_override = "PartnerAPI_Error_Events"
files = [ "/zoom/logs/events.log" ]
from_beginning = false
watch_method = "inotify"
[inputs.logparser.grok]
...

System info:

[Include Telegraf version, operating system name, and other relevant details]
Telegraf 1.6.0 on Redhat Linux

Steps to reproduce:

1. ...
2. ...

Expected behavior:

Actual behavior:

Additional info:

[Include gist of relevant config, logs, etc.]

[danielnelson]

Possibly related to #2847

[dgnorton]

Same or similar issue reported in the influxdb issue tracker: influxdata/influxdb#9664

[bolek2000]

After migration to telegraf 1.8 and tail plugin with grok data_format this issue seems to be resolved for me.

[bolek2000]

Still see this in 1.10 (with inotify and epoll), but not every time, this is kind of random behavior. Also sometimes it just ingests only some of the metrics.

[sjwang90]

@bolek2000 Is this a consistent problem in the latest Telegraf?

[bolek2000]

I am not sure if this problem is solved, because we built a workaround that is still in place. We have our own nginx access log preprocessing script, which sorts time slices of buffered logs chronologically and then appends the sorted lines to a new logfile, which gets ingested by inputs.tail plugin. But when we rotate/truncate the logfile we always do a systemctl reload telegraf, to make sure the rotation is always recognised by the agent/plugin. I'm not sure if I will have the time to test this again.

[pprkut]

We have integrated the telegraf restarting into an icinga check. Over a set of 100 servers, every day some of them need telegraf to be restarted because it stopped parsing the logs. We're on telegraf 1.15.3 now.

[pprkut]

Updated to 1.16.0 and still seeing log messages like

E! [inputs.logparser] Error stopping tail on file /var/log/secure

So the fixes in 1.16.0 at least don't fix this particular issue

[MyaLongmire]

Closing this as logparser is deprecated.