filebeat 1.2.2 reships rotated log files upon restart due to lost offset #1528
Comments
|
@jstangroome Thanks for the report. I will have a look and hopefully be able to provide a fix. |
|
I created a test case for the issue here: #1532 Hopefully I can provide a fix soon. |
|
@jstangroome I think I found a fix in #1532 |
ruflin
added a commit
to ruflin/beats
that referenced
this issue
May 2, 2016
Following changes were made * All state information is fetched in advance for later comparison to prevent race conditions * State is also written to registrar file if offset is 0, same for path * Test case to validate fix added
tsg
pushed a commit
that referenced
this issue
May 2, 2016
|
Closing as #1532 was merged. This fix will also be forwardported to master. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We just updated from filebeat 1.1.2 to 1.2.2 to resolve #1281 in our environment. This has revealed a new issue.
We are experiencing the problem on a local ext4 file system on Ubuntu 14.04 running in AWS EC2 and also running in Virtualbox 5.
I have created a bash script which reproduces the issue and published it as a gist, including a log of the output:
https://gist.github.com/jstangroome/0774df94b46aa288b0855f533c67d152
Essentially when filebeat exits, the registry is recorded with the entry for the fully-shipped and recently-rotated log file not having an offset specified.
When filebeat is started again, the missing offset value is treated effectively the same as
"offset": 0and the log file's contents are reshipped. Because the file was rotated recently, it is still new enough thatignore_olderdirectives won't exclude it./cc @ruflin
The text was updated successfully, but these errors were encountered: